3

I launched a wget job with the following command:

urls=(https://example.com/file1 https://example.com/file2
      https://example.com/file3 https://example.com/file4)
for url in ${urls[@]}; do 
  wget "$url" 2>/dev/null && echo "$url done" >> progress & 
done

These were files of ~30-40G each so I let it run overnight. I checked again today and found the following:

-rw-rw-r-- 1 a staff  13M Mar 18 15:11 file1
-rw-rw-r-- 1 a staff  18M Mar 18 15:12 file1.1
-rw-rw-r-- 1 a staff  38G Mar 18 21:10 file1.2
-rw-rw-r-- 1 a staff  12M Mar 18 15:12 file2
-rw-rw-r-- 1 a staff  44G Mar 18 21:35 file2.1
-rw-rw-r-- 1 a staff  446 Mar 18 21:35 progress
-rw-rw-r-- 1 a staff  40G Mar 18 21:24 file3.1
-rw-rw-r-- 1 a staff 4.4M Mar 18 15:12 file3
-rw-rw-r-- 1 a staff 6.5M Mar 18 15:12 file4
-rw-rw-r-- 1 a staff  42G Mar 18 21:27 file4.1

In other words, I see a few filename.N files, which look like they might be the right size, but the filename files themselves are clearly too small. The progress file reports that all files were downloaded successfully:

https://example.com/file1 done 
https://example.com/file2 done 
https://example.com/file3 done 
https://example.com/file4 done 

I have asked the person who made these available to me to provide me with md5sums of each file so I can see if any of the filename.N files are actually full copies. In the meantime, can I safely assume that the filename.N with the greatest value of N is actually a complete copy of the file in question?

I am guessing that the wget process was restarted a few times and that's why I have the .N files but given that the && echo seems to have worked, does that mean that the files were indeed, eventually, downloaded correctly or could the && have been executed even for an incomplete download?

  • I think that depends on how the server breaks the connection. I've had apparently-successful downloads from servers that stopped serving the file in the middle (possibly due to a buggy proxy in between). Unless you trust the web server and any proxy on the path, I recommend to obtain the file size through another channel. – Gilles Mar 20 '16 at 18:47
2

You could look at the headers sent by the server (--server-response), and if they include a Content-Length then wget will only exit when it receives that many bytes. This is mentioned in the man page for --ignore-length arg. But md5 or other sums are always a good idea!

1

My interpretation of https://www.gnu.org/software/wget/manual/wget.html

‘--no-clobber’ If a file is downloaded more than once in the same directory, Wget’s behavior depends on a few options, including ‘-nc’. In certain cases, the local file will be clobbered, or overwritten, upon repeated download. In other cases it will be preserved.

When running Wget without ‘-N’, ‘-nc’, ‘-r’, or ‘-p’, downloading the same file in the same directory will result in the original copy of file being preserved and the second copy being named ‘file.1’. If that file is downloaded yet again, the third copy will be named ‘file.2’, and so on.

... is that wget put the latest versions in the .N files and so can be trusted. The only other concern to rule out would an overriding WGETRC or other global wgetrc file.

  • Thanks, that's my interpretation as well but these are really quite important files for work and I don't want to take any chances, so I wanted a second opinion. What I'm really afraid of is whether there can be any cases where wget exits with an exit status of 0 but the file wasn't downloaded in its entirety. Do you have any idea if that could happen? – terdon Mar 20 '16 at 16:30
  • I would grab the src to the version you used and trace through it, in that case. I'll leave that to another answerer for now; sorry! – Jeff Schaller Mar 20 '16 at 16:33
  • 1
    Caveat: the server (or a proxy) might be buggy and serve a partial file but not give any indication of that. – Gilles Mar 20 '16 at 18:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.