0

I'm trying to use wget to download all Betacoronavirus files ending in .tar.gz from https://ftp.ncbi.nlm.nih.gov/blast/db/. I can use wget and paste in the link to a single one of those files, and it will download that just fine. But when I try to use some sort of regex, e.g. something with Betacoronavirus.*.tar.gz to download them all at once for me, it doesn't work. I've tried multiple things I've seen on some other threads, but still, none work.

1
  • Different method, similar outcome (compare: XY problem): looks like these files are available via FTP under ftp://ftp.ncbi.nlm.nih.gov/blast/db/, so you can use curlftpfs and work with them as if they were local (e.g. cd mountpoint && cp Betacoronavirus.*.tar.gz /somewhere/). Note this only works because the server provides FTP access. There is FUSE for HTTP(S), but since there is no notion of listable directories in HTTP, you cannot mount db/ and get files from it. May 22 at 18:50

2 Answers 2

3

Use a protocol designed for transferring files rather than text. In this case, the hostname "ftp" suggests you can also access the resources via FTP. wget can glob via FTP:

wget 'ftp://ftp.ncbi.nlm.nih.gov:21/blast/db/Betacoronavirus.*.tar.gz'
0

When you use "regex" or "glob" Betacoronavirus.*.tar.gz on the wget command-line, it is getting executed on your local machine in your Current Working Directory.

wget Betacoronavirus.*.tar.gz

In that Directory, there may be no such file and the "glob" is empty ; Alternatively, the Directory may contain some already downloaded file, then the "glob" may match that and try to download the same file.

Either way, that will not work.

What you require is a glob on the remote ftp machine. You can get that by using a cli ftp client.
Your best way (simple way) would be to use a gui ftp client and select the matching files and click download. I recommend WinSCP & FileZilla.

Advanced way would be to download the html source for the Directory Listing, Parse it and take out the matching files and execute wget on each matching file. I do not recommend this, given your use case.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.