2

I am downloading ~330k scientific files with wget from a csv file containing the URLs of the files I need to download. So, I am using the command

wget -i all_the_urls.csv

and my file all_the_urls.csv looks something like this

http://dr16.sdss.org/sas/dr16/sdss/spectro/redux/v5_13_0/spectra/lite/11040/spec-11040-58456-0109.fits
http://dr16.sdss.org/sas/dr16/sdss/spectro/redux/v5_13_0/spectra/lite/11068/spec-11068-58488-0780.fits
http://dr16.sdss.org/sas/dr16/sdss/spectro/redux/v5_13_0/spectra/lite/11068/spec-11068-58488-0240.fits
...

The files are small, ~250 kB each. But their amount is so large that it would take days to download them with wget, if it is just one file after the other. Is there a wget command to use so that I can download multiple files simultaneously? Note - my question is not duplicate to many other about downloading many files from the same website. Also, please don't suggest me wget2 - I am having some problems installing wget2 and here I look for other solutions.

It might be relevant, that my internet connection is ~550 MB/sec and when the files are downloaded 1 by 1 their download speed is displayed as 150 kB/sec, which is minuscule in comparison. So regarding bandwidth it would make sense to download many files at once (at least I guess so)

Can you also tell me how to execute one of this potential alternatives:

  • zip the files before I download them in the hopes that the single large zip file will be quicker to download
  • the files that I am downloading are .fits and contain 3 tables. I actually need only the first table in each of them. So if I can make wget access and download only the first table (preferably append all the 1st table into one big file/table), then this might save time, too

Tnx for any suggestions or ideas

1 Answer 1

3

You can use parallel. It's available in most Linux distros.

-jN means that N parallel programs will run. You can tune it to something like -j8 or even more.

cat url-list | parallel -j8 wget {}

or alternatively

cat url-list | parallel -j8 wget ' ' {}

cat is reading the file with URLs to download. My file looked liked:

https://XXX/all/Packages/a/abrt-2.10.9-20.el8.x86_64.rpm
https://XXX/all/Packages/a/abrt-addon-vmcore-2.10.9-20.el8.x86_64.rpm
...

The output is then piped to parallel that is responsible for spinning multiple wgets (-jN decide how many). The {} is piped input line.

7
  • 1
    Thanks a lot! Unfortunately I get the error /usr/bin/bash: wgethttp://dr16.sdss.org/sas/dr16/sdss/spectro/redux/v5_13_0/spectra/lite/9364/spec-9364-57699-0589.fits: No such file or directory. I guess your answer needs to add a space between wget and the URL. I am right now looking how to do this, but if you know please share, too! If you do this and say in a sentence what cat is I can accept your answer :)
    – NeStack
    Nov 24, 2021 at 17:32
  • I'm quite sure that there is space beetween wget and {}. Nov 24, 2021 at 17:41
  • For me I had to change your snippet cat url-list | parallel -j8 wget ' ' {} and then it worked like a charm. Strange. I accepted your answer and will include this alternative formulation in it. thanks again!
    – NeStack
    Nov 24, 2021 at 17:51
  • 2
    Use parallel -a url-list to avoid UUOC.
    – pLumo
    Nov 25, 2021 at 13:18
  • 2
    My first thought when I saw this answer was also "that use of cat is useless", I would have used parallel :::: url-list to avoid though, but that's a matter of style. Nov 25, 2021 at 13:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .