9

I have a lot of files I need sorting; and sadly there are many files with the same name but different content, and there are the same content with different filenames.

I'm thinking about using md5sum to generate checksums for the files, but I need to know - is it possible that two different files (ie. different content) will generate the same checksum?

If it is, how likely is it to happen?

Would it be possible to use two different unrelated (ie. not in the same "family") checksum-programs to generate two checksums - under the assumption that while two different files may generate the same checksum for either one of the checksum-programs, it would never happen to both at once?

  • 2
    Remember md5/sha2/etc is a HASH of data thus mapping a lot bits of information to a smaller bit space, thus it is possible there might be collisions. A HASH is not a unique key and if that is your goal...I'd strongly recommend to do something else. – mdpc May 22 '14 at 16:50
  • For sorting files, MD5 is fine. From crypto.stackexchange.com/a/18337/49945, probability of checksum collision of two different contents is 1 in 2^128 which is about 1 in 10^43. That's really, really rare. If you do one comparison per microsecond, on average you would get a collision once in 10^20 years. Its only been ~10^9 years since Big Bang, so the odds of a collision happening EVEN A SINGLE TIME, at one comparison per microsecond from Big Bang till now is still only 1 in 10^11. Given a hundred billion simultaneous computers, only ~one would have seen a collision yet. – ToolmakerSteve Jul 24 '17 at 11:00
  • ... though it isn't clear why you would use any hash algorithm when sorting files. If there are files with same name, they must be identifiable by some other criteria; e.g. there must be some unique path to the file, or perhaps a byte offset within some storage device? Either of those would be a more useful way to uniquely refer to the file. – ToolmakerSteve Jul 24 '17 at 11:11
10

Two files with same contents but different filenames: (file1 and file2):

cat file1
this is a simple file

cat file2
this is a simple file

md5sum file1
7de45bf879db49de7e2eacea23e6c165  file1
md5sum file2
7de45bf879db49de7e2eacea23e6c165  file2

Two files with different contents but same filenames: (file1 and file1)

cat file1
this is a simple file
cat file1
this is a simple file with extra contents
md5sum file1 #first file1
7de45bf879db49de7e2eacea23e6c165  file1
md5sum file1 #second file1
c7c8f3fd9ddd7a926c31416a69063e4e  file1

From the wiki entry,

However, it is very unlikely that any two non-identical files in the real world will have the same MD5 hash, unless they have been specifically created to have the same hash.

But, the MD5 algorithm has it's own flaws.

However, now that it is easy to generate MD5 collisions, it is possible for the person who created the file to create a second file with the same checksum, so this technique cannot protect against some forms of malicious tampering. Also, in some cases, the checksum cannot be trusted (for example, if it was obtained over the same channel as the downloaded file), in which case MD5 can only provide error-checking functionality: it will recognize a corrupt or incomplete download, which becomes more likely when downloading larger files.

I would recommend using sha1 for calculating the checksum since producing collisions is not that easy when using sha1 algorithms. Producing sha1 checksum is pretty easy as you can see here.

  • NOTE: the question is about sorting files; the recommendation to use sha1 is not relevant to that task; it only matters when concerned about security. – ToolmakerSteve Jul 24 '17 at 11:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.