Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm wget'ing huge files (over 100GB) and then comparing my md5sum to the sum posted in the repository. Believe it or not, md5sum takes a nontrivial amount of time to run on a file of this size, so I'd like to get the sum in parallel with the download.

What's the best way to do this? I don't see a wget option to calculate MD5 during download (surprisingly) so I guess using tee would be the way to go?!

share|improve this question
Is this possible? I'm pretty sure you need the entire binary to check the hash. I doubt even breaking it into steps (ie hash as it is downloaded) would be possible seeing as you might not be receiving the file in sequential order. Either way, I would be happy to see if someone has a solution. – n0pe Aug 20 '11 at 2:06
@MaxMackie: It is definitely possible: since I'm downloading the entire file, wget could theoretically run it through MD5. Unf, it doesn't. – Fixee Aug 20 '11 at 2:29
up vote 8 down vote accepted

Actually, this might be possible:

wget -O - http://example.com/file | tee file | md5sum > file.md5

However, this assumes you have the hash in a file beforehand.

share|improve this answer
This looks fine, and I don't see how it assumes anything about having the "has in a file beforehand". It just generates the sum as you go along and writes it to a file that you can use to compare later. The only catch might be the memory overhead. – Caleb Aug 20 '11 at 6:10
@Caleb What memory overhead? md5sum (and any other common hash algorithm such as the SHA family) operates in constant memory. They're designed to work on streams. – Gilles Aug 21 '11 at 1:27
with an additional awk at the end you can remove the ` -` a the end automatically. wget -O - "https://example.org/myFile.jpg" | tee file | md5sum | awk '{print $1}' > md5.txt – Max Dec 11 '14 at 13:38

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.