12

I'm writing a script that involves downloading and manipulating a file, and I want to ensure that the file is not incomplete (due to e.g. a dropped connection) before I work on it.

10

The most common way to verify the integrity of downloaded files is to use MD5 checksums. This assumes that the site you are downloading from actually published MD5 checksums of their files. You can verify a MD5 checksum by creating your own checksum of the downloaded file and comparing it to the published checksum. If they are identical the file you have downloaded is complete and not tampered with.

If you don't expect the file you are downloading to change you can precompute a checksum and hard code it into the script, but if the file is ever updated the verification will fail.

To create a MD5 checksum of a file run md5sum myFile. In the case of wget you might find this command useful, especially if the file you are downloading is large:

wget -O - http://example.com/myFile | tee myFile | md5sum > MD5SUM.

This will create a checksum of "myFile" while downloading and save it to the file MD5SUM, possibly saving you some time.

In the case of a dropped connection I think the best way would be to check the exit codes of wget. If the download is successful without any errors wget will return 0. Anything else indicates something went wrong. Take a look at the "Exit status" section of man wget.

7

The return code of the command used to download the file will tell you if the command executed successfully or not. Typically, a return code of 0 denotes success and any non-zero number denotes an error. You can access the return code through the $? variable.

A basic example using wget would go:

#!/bin/bash

wget foo.tgz &> /dev/null

if [[ "$?" != 0 ]]; then
    echo "Error downloading file"
else
    echo "Success"
fi

&> /dev/null redirects all of wget's output to /dev/null so it's ideal for scripting BUT it's makes debugging wget errors more difficult.

  • 4
    you might do instead: wget -q ... || { handle ; error ; } – mikeserv Jun 7 '14 at 13:55
  • @mikeserv Didn't even know that was there, nice touch – Creek Jun 7 '14 at 13:58
  • 1
    only found it in man while reading through to answer and the two things I was going to say were already here in two answers - so I did two comments. Nice touch to you, too. – mikeserv Jun 7 '14 at 13:59
  • I expect this to not work when used with SOCKS proxies like tor. – CodesInChaos Jun 7 '14 at 17:47
  • 1
    @Creek What I meant is that wget might think the download was complete, even if it broke. Proxies information about broken TCP connections vs. closed TCP connections which is problematic with HTTP since it uses closed TCP as end mark by default. That's why I added a check if the file size from the header matches the downloaded file size when I ran a mass download. Not sure if wget does such consistency checks or what the http spec says on this issue. – CodesInChaos Jun 7 '14 at 18:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.