I would like to use wget recursively on an ftp directory like this:


Which contains another 2 directory levels inside, the first with the species name, and one inside with the name dna, like this: ftp://ftp.ensembl.org/pub/current_fasta/species_name/dna/

For example, one of them has the species_name homo_sapiens:


Inside each ./dna/ directory, I would like to download only the file with the name containing the word toplevel but not the one containing the word _rm.toplevel.

Finally, I would like a wget recursive command that would download all those from these directories, each as an output file with the name species_name.fa.gz, where the species_name is the name of the parent directory, just before dna.

Any ideas?


You can tell wget to include or exclude files and directories based on wildcard patterns. Untested:

wget -R \
  -X '/*/?*dna,/*/pep' \
  -A '*toplevel*' -R '*_rm.toplevel*' \

For FTP, wget isn't the best tool for complex retrieval. Wget knows how to follow links in a web page, but that's not relevant for FTP. I recommend mounting the FTP site as a directory with AVFS, CurlFtpFS or some other FUSE filesystem, and using cp or rsync. With AVFS:

ls ~/.avfs/\#ftp:ftp.ensembl.org/pub/current_fasta/homo_sapiens

With CurlFtpFS:

mkdir ~/current_fasta
curlftpfs ftp://ftp.ensembl.org/pub/current_fasta/ ~/current_fasta
ls ~/current_fasta/homo_sapiens

To copy the files in any shell:

for x in ~/.avfs/\#ftp:ftp.ensembl.org/pub/current_fasta/*/dna/*toplevel*; do
  case $x in *_rm.toplevel*) continue;; esac
  species=${x%/dna/*}; species=${species##*/}
  cp -p "$x" "$species.fa.gz"

Or in zsh:

autoload zmv
zmv -C '~/.avfs/\#ftp:ftp.ensembl.org/pub/current_fasta/(*)/dna/*toplevel*~*_rm.toplevel*' '$1.fa.gz'

If you can use lftp:

echo 'mirror --no-empty-dirs -I *toplevel* -X *_rm.toplevel* . .' | 
lftp ftp://ftp.ensembl.org/pub/current_fasta/
for folder in *; do
  mv $folder/dna/*toplevel* $folder.fa.gz
  rm -r "$folder"
  • Excellent, thanks. I only change the renaming loop for a find/awk combo like this ($8 counts number of dirs): find pwd -name "*.gz" | sort | grep toplevel | awk -F\/ '{print "mv "$i" "$8".fa.gz"}' > name.sh && bash ./name.sh
    – 719016
    Jul 29 '11 at 13:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.