2

Consider a folder with many XML files (10K small text files). Some XML files are identical, some are different.

I would like to find out what files are identical (ignoring whitespaces, tabs and linebreaks) and record the files in each cluster somehow.

I don't need high precision on this, so I thought one way of doing this would be with MD5 or any other hashing algorirhm, i.e. count the number of files with the same exact MD5 sum, but I would need to pre-remove spaces.

I'm in OS X and can check the MD5 of a file as follows:

$ md5 file_XYZ.xml
MD5 (file_XYZ.xml) = 0de0c7bea1a75434934c3821dcba759a

How can I use this to cluster identical files? (either a text file with filenames with the same hash, or clustering files in folders would do it)

  • 1
    how about fdupes? It takes care of all of that for you. – SailorCire May 6 '15 at 17:51
2

You could create a "normalized" version of each XML file with something like:

xmllint --nospace --format orginal.xml > normalized.xml

That would get rid of "unimportant"-to-XML whitespace, indent consistently and so forth. After that, you could use cksum to find identical normalized files.

I'll suggest a script:

for ORIGXML in *.xml
do
    xmllint --noblank --format "$ORIGXML" > "normalized.$ORIGXML"
    cksum "normalized.$ORIGXML" | sed 's/^normalized\.//' >> files.list
done
sort -k1.1 files.list > sorted.files

I'm not sure I'd bother with an MD5 checksum. You're looking for duplicates, not doing cryptography with evil adversaries opposing you.

If you're looking for "nearly identical" XML files, you could maybe use Normalized Compression Distance to see how "far apart" the files are from each other. More simply, you could gzip or bzip2 the XML files, then sort based on compressed file size. The closer the compressed file size, the more identical the XML files will be.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.