Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I would like to use wget recursively on an ftp directory like this:


Which contains another 2 directory levels inside, the first with the species name, and one inside with the name dna, like this: ftp://ftp.ensembl.org/pub/current_fasta/species_name/dna/

For example, one of them has the species_name homo_sapiens:


Inside each ./dna/ directory, I would like to download only the file with the name containing the word toplevel but not the one containing the word _rm.toplevel.

Finally, I would like a wget recursive command that would download all those from these directories, each as an output file with the name species_name.fa.gz, where the species_name is the name of the parent directory, just before dna.

Any ideas?

share|improve this question
up vote 3 down vote accepted

If you can use lftp:

echo 'mirror --no-empty-dirs -I *toplevel* -X *_rm.toplevel* . .' | 
lftp ftp://ftp.ensembl.org/pub/current_fasta/
for folder in *; do
  mv $folder/dna/*toplevel* $folder.fa.gz
  rm -r "$folder"
share|improve this answer
Excellent, thanks. I only change the renaming loop for a find/awk combo like this ($8 counts number of dirs): find pwd -name "*.gz" | sort | grep toplevel | awk -F\/ '{print "mv "$i" "$8".fa.gz"}' > name.sh && bash ./name.sh – 719016 Jul 29 '11 at 13:11

You can tell wget to include or exclude files and directories based on wildcard patterns. Untested:

wget -R \
  -X '/*/?*dna,/*/pep' \
  -A '*toplevel*' -R '*_rm.toplevel*' \

For FTP, wget isn't the best tool for complex retrieval. Wget knows how to follow links in a web page, but that's not relevant for FTP. I recommend mounting the FTP site as a directory with AVFS, CurlFtpFS or some other FUSE filesystem, and using cp or rsync. With AVFS:

ls ~/.avfs/\#ftp:ftp.ensembl.org/pub/current_fasta/homo_sapiens

With CurlFtpFS:

mkdir ~/current_fasta
curlftpfs ftp://ftp.ensembl.org/pub/current_fasta/ ~/current_fasta
ls ~/current_fasta/homo_sapiens

To copy the files in any shell:

for x in ~/.avfs/\#ftp:ftp.ensembl.org/pub/current_fasta/*/dna/*toplevel*; do
  case $x in *_rm.toplevel*) continue;; esac
  species=${x%/dna/*}; species=${species##*/}
  cp -p "$x" "$species.fa.gz"

Or in zsh:

autoload zmv
zmv -C '~/.avfs/\#ftp:ftp.ensembl.org/pub/current_fasta/(*)/dna/*toplevel*~*_rm.toplevel*' '$1.fa.gz'
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.