I'm trying to use wget to download all Betacoronavirus files ending in .tar.gz from https://ftp.ncbi.nlm.nih.gov/blast/db/. I can use wget and paste in the link to a single one of those files, and it will download that just fine. But when I try to use some sort of regex, e.g. something with Betacoronavirus.*.tar.gz to download them all at once for me, it doesn't work. I've tried multiple things I've seen on some other threads, but still, none work.
2 Answers
Use a protocol designed for transferring files rather than text. In this case, the hostname "ftp" suggests you can also access the resources via FTP. wget can glob via FTP:
wget 'ftp://ftp.ncbi.nlm.nih.gov:21/blast/db/Betacoronavirus.*.tar.gz'
When you use "regex" or "glob" Betacoronavirus.*.tar.gz on the wget command-line, it is getting executed on your local machine in your Current Working Directory.
wget Betacoronavirus.*.tar.gz
In that Directory, there may be no such file and the "glob" is empty ; Alternatively, the Directory may contain some already downloaded file, then the "glob" may match that and try to download the same file.
Either way, that will not work.
What you require is a glob on the remote ftp machine. You can get that by using a cli ftp client.
Your best way (simple way) would be to use a gui ftp client and select the matching files and click download. I recommend WinSCP & FileZilla.
Advanced way would be to download the html source for the Directory Listing, Parse it and take out the matching files and execute wget on each matching file. I do not recommend this, given your use case.
ftp://ftp.ncbi.nlm.nih.gov/blast/db/, so you can usecurlftpfsand work with them as if they were local (e.g.cd mountpoint && cp Betacoronavirus.*.tar.gz /somewhere/). Note this only works because the server provides FTP access. There is FUSE for HTTP(S), but since there is no notion of listable directories in HTTP, you cannot mountdb/and get files from it.