I'm wget'ing huge files (over 100GB) and then comparing my md5sum to the sum posted in the repository. Believe it or not, md5sum takes a nontrivial amount of time to run on a file of this size, so I'd like to get the sum in parallel with the download.

What's the best way to do this? I don't see a wget option to calculate MD5 during download (surprisingly) so I guess using tee would be the way to go?!

  • Is this possible? I'm pretty sure you need the entire binary to check the hash. I doubt even breaking it into steps (ie hash as it is downloaded) would be possible seeing as you might not be receiving the file in sequential order. Either way, I would be happy to see if someone has a solution. – n0pe Aug 20 '11 at 2:06
  • @MaxMackie: It is definitely possible: since I'm downloading the entire file, wget could theoretically run it through MD5. Unf, it doesn't. – Fixee Aug 20 '11 at 2:29

Actually, this might be possible:

wget -O - http://example.com/file | tee file | md5sum > file.md5

However, this assumes you have the hash in a file beforehand.

  • 1
    This looks fine, and I don't see how it assumes anything about having the "has in a file beforehand". It just generates the sum as you go along and writes it to a file that you can use to compare later. The only catch might be the memory overhead. – Caleb Aug 20 '11 at 6:10
  • 2
    @Caleb What memory overhead? md5sum (and any other common hash algorithm such as the SHA family) operates in constant memory. They're designed to work on streams. – Gilles Aug 21 '11 at 1:27
  • with an additional awk at the end you can remove the ` -` a the end automatically. wget -O - "https://example.org/myFile.jpg" | tee file | md5sum | awk '{print $1}' > md5.txt – Max Dec 11 '14 at 13:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.