12

I'm wget'ing huge files (over 100GB) and then comparing my md5sum to the sum posted in the repository. Believe it or not, md5sum takes a nontrivial amount of time to run on a file of this size, so I'd like to get the sum in parallel with the download.

What's the best way to do this? I don't see a wget option to calculate MD5 during download (surprisingly) so I guess using tee would be the way to go?!

2
  • Is this possible? I'm pretty sure you need the entire binary to check the hash. I doubt even breaking it into steps (ie hash as it is downloaded) would be possible seeing as you might not be receiving the file in sequential order. Either way, I would be happy to see if someone has a solution.
    – nopcorn
    Aug 20, 2011 at 2:06
  • @MaxMackie: It is definitely possible: since I'm downloading the entire file, wget could theoretically run it through MD5. Unf, it doesn't.
    – Fixee
    Aug 20, 2011 at 2:29

1 Answer 1

13

Actually, this might be possible:

wget -O - http://example.com/file | tee file | md5sum > file.md5

However, this assumes you have the hash in a file beforehand.

3
  • 1
    This looks fine, and I don't see how it assumes anything about having the "has in a file beforehand". It just generates the sum as you go along and writes it to a file that you can use to compare later. The only catch might be the memory overhead.
    – Caleb
    Aug 20, 2011 at 6:10
  • 3
    @Caleb What memory overhead? md5sum (and any other common hash algorithm such as the SHA family) operates in constant memory. They're designed to work on streams. Aug 21, 2011 at 1:27
  • with an additional awk at the end you can remove the ` -` a the end automatically. wget -O - "https://example.org/myFile.jpg" | tee file | md5sum | awk '{print $1}' > md5.txt
    – Max
    Dec 11, 2014 at 13:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .