2

I am downloading ~330k scientific files with wget from a csv file containing the URLs of the files I need to download. So, I am using the command

wget -i all_the_urls.csv

and my file all_the_urls.csv looks something like this

http://dr16.sdss.org/sas/dr16/sdss/spectro/redux/v5_13_0/spectra/lite/11040/spec-11040-58456-0109.fits
http://dr16.sdss.org/sas/dr16/sdss/spectro/redux/v5_13_0/spectra/lite/11068/spec-11068-58488-0780.fits
http://dr16.sdss.org/sas/dr16/sdss/spectro/redux/v5_13_0/spectra/lite/11068/spec-11068-58488-0240.fits
...

The files are small, ~250 kB each. But their amount is so large that it would take days to download them with wget, if it is just one file after the other. Is there a wget command to use so that I can download multiple files simultaneously? Note - my question is not duplicate to many other about downloading many files from the same website. Also, please don't suggest me wget2 - I am having some problems installing wget2 and here I look for other solutions.

It might be relevant, that my internet connection is ~550 MB/sec and when the files are downloaded 1 by 1 their download speed is displayed as 150 kB/sec, which is minuscule in comparison. So regarding bandwidth it would make sense to download many files at once (at least I guess so)

Can you also tell me how to execute one of this potential alternatives:

  • zip the files before I download them in the hopes that the single large zip file will be quicker to download
  • the files that I am downloading are .fits and contain 3 tables. I actually need only the first table in each of them. So if I can make wget access and download only the first table (preferably append all the 1st table into one big file/table), then this might save time, too

Tnx for any suggestions or ideas

2

You can use parallel. It's available in most Linux distros.

-jN means that N parallel programs will run. You can tune it to something like -j8 or even more.

cat url-list | parallel -j8 wget {}

or alternatively

cat url-list | parallel -j8 wget ' ' {}

cat is reading the file with URLs to download. My file looked liked:

https://XXX/all/Packages/a/abrt-2.10.9-20.el8.x86_64.rpm
https://XXX/all/Packages/a/abrt-addon-vmcore-2.10.9-20.el8.x86_64.rpm
...

The output is then piped to parallel that is responsible for spinning multiple wgets (-jN decide how many). The {} is piped input line.

7
  • 1
    Thanks a lot! Unfortunately I get the error /usr/bin/bash: wgethttp://dr16.sdss.org/sas/dr16/sdss/spectro/redux/v5_13_0/spectra/lite/9364/spec-9364-57699-0589.fits: No such file or directory. I guess your answer needs to add a space between wget and the URL. I am right now looking how to do this, but if you know please share, too! If you do this and say in a sentence what cat is I can accept your answer :)
    – NeStack
    Nov 24 at 17:32
  • I'm quite sure that there is space beetween wget and {}. Nov 24 at 17:41
  • For me I had to change your snippet cat url-list | parallel -j8 wget ' ' {} and then it worked like a charm. Strange. I accepted your answer and will include this alternative formulation in it. thanks again!
    – NeStack
    Nov 24 at 17:51
  • 1
    Use parallel -a url-list to avoid UUOC.
    – pLumo
    Nov 25 at 13:18
  • 2
    My first thought when I saw this answer was also "that use of cat is useless", I would have used parallel :::: url-list to avoid though, but that's a matter of style. Nov 25 at 13:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.