Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I would like to use wget recursively on an ftp directory like this:

ftp://ftp.ensembl.org/pub/current_fasta/

Which contains another 2 directory levels inside, the first with the species name, and one inside with the name dna, like this: ftp://ftp.ensembl.org/pub/current_fasta/species_name/dna/

For example, one of them has the species_name homo_sapiens:

ftp://ftp.ensembl.org/pub/current_fasta/homo_sapiens/dna/

Inside each ./dna/ directory, I would like to download only the file with the name containing the word toplevel but not the one containing the word _rm.toplevel.

Finally, I would like a wget recursive command that would download all those from these directories, each as an output file with the name species_name.fa.gz, where the species_name is the name of the parent directory, just before dna.

Any ideas?

share|improve this question

2 Answers 2

up vote 3 down vote accepted

If you can use lftp:

echo 'mirror --no-empty-dirs -I *toplevel* -X *_rm.toplevel* . .' | 
lftp ftp://ftp.ensembl.org/pub/current_fasta/
for folder in *; do
  mv $folder/dna/*toplevel* $folder.fa.gz
  rm -r "$folder"
done
share|improve this answer
    
Excellent, thanks. I only change the renaming loop for a find/awk combo like this ($8 counts number of dirs): find pwd -name "*.gz" | sort | grep toplevel | awk -F\/ '{print "mv "$i" "$8".fa.gz"}' > name.sh && bash ./name.sh –  130490868091234 Jul 29 '11 at 13:11

You can tell wget to include or exclude files and directories based on wildcard patterns. Untested:

wget -R \
  -X '/*/?*dna,/*/pep' \
  -A '*toplevel*' -R '*_rm.toplevel*' \
  ftp://ftp.ensembl.org/pub/current_fasta/

For FTP, wget isn't the best tool for complex retrieval. Wget knows how to follow links in a web page, but that's not relevant for FTP. I recommend mounting the FTP site as a directory with AVFS, CurlFtpFS or some other FUSE filesystem, and using cp or rsync. With AVFS:

mountavfs
ls ~/.avfs/\#ftp:ftp.ensembl.org/pub/current_fasta/homo_sapiens

With CurlFtpFS:

mkdir ~/current_fasta
curlftpfs ftp://ftp.ensembl.org/pub/current_fasta/ ~/current_fasta
ls ~/current_fasta/homo_sapiens

To copy the files in any shell:

for x in ~/.avfs/\#ftp:ftp.ensembl.org/pub/current_fasta/*/dna/*toplevel*; do
  case $x in *_rm.toplevel*) continue;; esac
  species=${x%/dna/*}; species=${species##*/}
  cp -p "$x" "$species.fa.gz"
done

Or in zsh:

autoload zmv
zmv -C '~/.avfs/\#ftp:ftp.ensembl.org/pub/current_fasta/(*)/dna/*toplevel*~*_rm.toplevel*' '$1.fa.gz'
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.