0

I am compressing a number of files with size in the order of 5GB before sharing them.

On the one hand, I would like to provide a hash sum for the recipient to check the files' integrity. On the other hand, generating a sha256sum of a large file takes some time if you don't have a powerful machine and I did not want to overburden the recipients' computer.

I then wondered whether a faster sha1 sum would provide the same integrity guarantees as a slower sha256sum, simply because the files are big enough.

Take note that security is not a concern here: mainly, it is about knowing that the data have not been accidentally corrupted during upload, storage and download.

Question stimulated by this other answer https://unix.stackexchange.com/a/260519/132913

Improve this question
1
  • fwiw, shattered.it with two different valid PDF with the same size, the same sha1 hash but different content: PDF1 PDF2.
    – A.B
    Nov 10, 2020 at 14:33

1 Answer 1

Reset to default
1

Take note that security is not a concern here: mainly, it is about knowing that the data have not been accidentally corrupted during upload, storage and download.

If that's not a concern there's no need to use more advanced and CPU intensive hashing algos. The possibility of a random transmission bit (or several bits) failure generating the same md5 hash is pretty much zero. E.g. even CRC32 might be enough in this case.

Personally when I need to send someone data and I have to be sure it's going to be sent without errors, I compress using RAR and add a recovery (1%) record. There are no native Unix utilities for that but you can combine compressing with sending par2 recovery data but we're talking about multiple files.

Improve this answer
4
  • To calculate the crc you can use the cksum or crc32 commands. they are calculating both crc32 but have a different output format. This is even faster than any hashing algortihm but also very easy to crack as a MITM. (ethernet is using crc32 to test if the transmission of a package was correct)
    – blaimi
    Nov 10, 2020 at 12:33
  • What is a recovery record? (Note as addition that I will not put the files in an archive, because they are many and bulky; I will put them in a repository for future users to pick the one of their choice.)
    – XavierStuvw
    Nov 10, 2020 at 14:08
  • google.com/search?q=what+is+rar+recovery+record
    – Artem S. Tashkinov
    Nov 10, 2020 at 14:09
  • Also superuser.com/questions/1362786/…
    – XavierStuvw
    Nov 10, 2020 at 15:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .