0

I am compressing a number of files with size in the order of 5GB before sharing them.

On the one hand, I would like to provide a hash sum for the recipient to check the files' integrity. On the other hand, generating a sha256sum of a large file takes some time if you don't have a powerful machine and I did not want to overburden the recipients' computer.

I then wondered whether a faster sha1 sum would provide the same integrity guarantees as a slower sha256sum, simply because the files are big enough.

Take note that security is not a concern here: mainly, it is about knowing that the data have not been accidentally corrupted during upload, storage and download.


Question stimulated by this other answer https://unix.stackexchange.com/a/260519/132913

1
  • fwiw, shattered.it with two different valid PDF with the same size, the same sha1 hash but different content: PDF1 PDF2.
    – A.B
    Nov 10 '20 at 14:33
1

Take note that security is not a concern here: mainly, it is about knowing that the data have not been accidentally corrupted during upload, storage and download.

If that's not a concern there's no need to use more advanced and CPU intensive hashing algos. The possibility of a random transmission bit (or several bits) failure generating the same md5 hash is pretty much zero. E.g. even CRC32 might be enough in this case.

Personally when I need to send someone data and I have to be sure it's going to be sent without errors, I compress using RAR and add a recovery (1%) record. There are no native Unix utilities for that but you can combine compressing with sending par2 recovery data but we're talking about multiple files.

4
  • To calculate the crc you can use the cksum or crc32 commands. they are calculating both crc32 but have a different output format. This is even faster than any hashing algortihm but also very easy to crack as a MITM. (ethernet is using crc32 to test if the transmission of a package was correct)
    – blaimi
    Nov 10 '20 at 12:33
  • What is a recovery record? (Note as addition that I will not put the files in an archive, because they are many and bulky; I will put them in a repository for future users to pick the one of their choice.) Nov 10 '20 at 14:08
  • Also superuser.com/questions/1362786/… Nov 10 '20 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.