Tell me more ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 4 down vote favorite

I'm wget'ing huge files (over 100GB) and then comparing my md5sum to the sum posted in the repository. Believe it or not, md5sum takes a nontrivial amount of time to run on a file of this size, so I'd like to get the sum in parallel with the download.

What's the best way to do this? I don't see a wget option to calculate MD5 during download (surprisingly) so I guess using tee would be the way to go?!

share|improve this question
Is this possible? I'm pretty sure you need the entire binary to check the hash. I doubt even breaking it into steps (ie hash as it is downloaded) would be possible seeing as you might not be receiving the file in sequential order. Either way, I would be happy to see if someone has a solution. – MaxMackie Aug 20 '11 at 2:06
@MaxMackie: It is definitely possible: since I'm downloading the entire file, wget could theoretically run it through MD5. Unf, it doesn't. – Fixee Aug 20 '11 at 2:29

1 Answer

up vote 4 down vote accepted

Actually, this might be possible:

wget -O - http://example.com/file | tee file | md5sum > file.md5

However, this assumes you have the hash in a file beforehand.

share|improve this answer
Yes, something like this. I think the last > sign is an error, however (you can't redirect stdout twice). – Fixee Aug 20 '11 at 2:35
@Fixee, sounds like you already have this figured out. I agree with you about tee: I think the end of MaxMackie's command should be tee file | md5sum > file.md5. – Klox Aug 20 '11 at 3:29
1  
This looks fine, and I don't see how it assumes anything about having the "has in a file beforehand". It just generates the sum as you go along and writes it to a file that you can use to compare later. The only catch might be the memory overhead. – Caleb Aug 20 '11 at 6:10
1  
@Caleb What memory overhead? md5sum (and any other common hash algorithm such as the SHA family) operates in constant memory. They're designed to work on streams. – Gilles Aug 21 '11 at 1:27

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.