I am downloading ~330k scientific files with wget from a csv file containing the URLs of the files I need to download. So, I am using the command
wget -i all_the_urls.csv
and my file all_the_urls.csv looks something like this
http://dr16.sdss.org/sas/dr16/sdss/spectro/redux/v5_13_0/spectra/lite/11040/spec-11040-58456-0109.fits http://dr16.sdss.org/sas/dr16/sdss/spectro/redux/v5_13_0/spectra/lite/11068/spec-11068-58488-0780.fits http://dr16.sdss.org/sas/dr16/sdss/spectro/redux/v5_13_0/spectra/lite/11068/spec-11068-58488-0240.fits ...
The files are small, ~250 kB each. But their amount is so large that it would take days to download them with
wget, if it is just one file after the other. Is there a wget command to use so that I can download multiple files simultaneously? Note - my question is not duplicate to many other about downloading many files from the same website. Also, please don't suggest me
wget2 - I am having some problems installing
wget2 and here I look for other solutions.
It might be relevant, that my internet connection is ~550 MB/sec and when the files are downloaded 1 by 1 their download speed is displayed as 150 kB/sec, which is minuscule in comparison. So regarding bandwidth it would make sense to download many files at once (at least I guess so)
Can you also tell me how to execute one of this potential alternatives:
- zip the files before I download them in the hopes that the single large zip file will be quicker to download
- the files that I am downloading are
.fitsand contain 3 tables. I actually need only the first table in each of them. So if I can make
wgetaccess and download only the first table (preferably append all the 1st table into one big file/table), then this might save time, too
Tnx for any suggestions or ideas