wget recursive for only files that pass a regexp

Question

I would like to use wget recursively on an ftp directory like this:

ftp://ftp.ensembl.org/pub/current_fasta/

Which contains another 2 directory levels inside, the first with the species name, and one inside with the name dna, like this: ftp://ftp.ensembl.org/pub/current_fasta/species_name/dna/

For example, one of them has the species_name homo_sapiens:

ftp://ftp.ensembl.org/pub/current_fasta/homo_sapiens/dna/

Inside each ./dna/ directory, I would like to download only the file with the name containing the word toplevel but not the one containing the word _rm.toplevel.

Finally, I would like a wget recursive command that would download all those from these directories, each as an output file with the name species_name.fa.gz, where the species_name is the name of the parent directory, just before dna.

Any ideas?

Answer 1

If you can use lftp:

echo 'mirror --no-empty-dirs -I *toplevel* -X *_rm.toplevel* . .' | 
lftp ftp://ftp.ensembl.org/pub/current_fasta/
for folder in *; do
  mv $folder/dna/*toplevel* $folder.fa.gz
  rm -r "$folder"
done

Answer 2

You can tell wget to include or exclude files and directories based on wildcard patterns. Untested:

wget -R \
  -X '/*/?*dna,/*/pep' \
  -A '*toplevel*' -R '*_rm.toplevel*' \
  ftp://ftp.ensembl.org/pub/current_fasta/

For FTP, wget isn't the best tool for complex retrieval. Wget knows how to follow links in a web page, but that's not relevant for FTP. I recommend mounting the FTP site as a directory with AVFS, CurlFtpFS or some other FUSE filesystem, and using cp or rsync. With AVFS:

mountavfs
ls ~/.avfs/\#ftp:ftp.ensembl.org/pub/current_fasta/homo_sapiens

With CurlFtpFS:

mkdir ~/current_fasta
curlftpfs ftp://ftp.ensembl.org/pub/current_fasta/ ~/current_fasta
ls ~/current_fasta/homo_sapiens

To copy the files in any shell:

for x in ~/.avfs/\#ftp:ftp.ensembl.org/pub/current_fasta/*/dna/*toplevel*; do
  case $x in *_rm.toplevel*) continue;; esac
  species=${x%/dna/*}; species=${species##*/}
  cp -p "$x" "$species.fa.gz"
done

Or in zsh:

autoload zmv
zmv -C '~/.avfs/\#ftp:ftp.ensembl.org/pub/current_fasta/(*)/dna/*toplevel*~*_rm.toplevel*' '$1.fa.gz'