0

I am downloading ~330k scientific files with wget from a csv file containing the URLs of the files I need to download. So, I am using the command

wget -i all_the_urls.csv

and my file all_the_urls.csv looks something like this

http://dr16.sdss.org/sas/dr16/sdss/spectro/redux/v5_13_0/spectra/lite/11040/spec-11040-58456-0109.fits
http://dr16.sdss.org/sas/dr16/sdss/spectro/redux/v5_13_0/spectra/lite/11068/spec-11068-58488-0780.fits
http://dr16.sdss.org/sas/dr16/sdss/spectro/redux/v5_13_0/spectra/lite/11068/spec-11068-58488-0240.fits

The files are small, ~ 250 kB each. But their amount is so large that I will need to sometimes interrupt the download and then continue the download again. Is there a wget command to use so that I can continue the download where I left of, after the last downloaded file, not at the beginning of the list of URLs?

If there is no such command can you tell me how I can use one of the following work-arounds (I am not aware of wget):

  • after wget downloads a file make it erase the URL of this file inside all_the_urls.csv
  • before wget downloads a new file make it check if this file is already downloaded and if it is, skip to the next file (not sure how time expensive this checking will be though)

Any other ideas are also welcome! tnx

Improve this question

1 Answer 1

Reset to default
1

-c for continue & possible -nc for no clobber should work.

You may be better to look at mirror although the docs on that are poor.

Improve this answer
4
  • Thank you, it looks like -nc is what I am looking for, it starts right after the file where I left of! -c on the other hand does some long checks and takes ~1sec/file, so it is definitely not an option for my case with 300k files
    – NeStack
    Nov 23, 2021 at 20:13
  • There is a possibility then that a file may only be partially downloaded. If you can, I would see about some form of a crc listing of the directory contents.
    – Bib
    Nov 23, 2021 at 20:16
  • @NeStack, just check what happens for the partial file if you interrupt the download mid-file. At least the wget I looked at happily leaves the partial file with the correct filename, and after that -nc ignores it. -c at least continues the download, since like you said, it checks the file size from the server, for each file.
    – ilkkachu
    Nov 23, 2021 at 20:22
  • An alternative would be to create a file listing, iterate through it, then delete on a successful transfer. Or add it to a done list and check there.
    – Bib
    Nov 23, 2021 at 20:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.