Consider a folder with many XML files (10K small text files). Some XML files are identical, some are different.
I would like to find out what files are identical (ignoring whitespaces, tabs and linebreaks) and record the files in each cluster somehow.
I don't need high precision on this, so I thought one way of doing this would be with MD5 or any other hashing algorirhm, i.e. count the number of files with the same exact MD5 sum, but I would need to pre-remove spaces.
I'm in OS X and can check the MD5 of a file as follows:
$ md5 file_XYZ.xml MD5 (file_XYZ.xml) = 0de0c7bea1a75434934c3821dcba759a
How can I use this to cluster identical files? (either a text file with filenames with the same hash, or clustering files in folders would do it)