Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

In Unix, is there any way to compare every text file in a directory to every other text file in the directory, and then sort each pair of files by similarity (using the diff utility)? There are already some command-line Unix programs (such as fdupes) that can find duplicate files in a directory, but I'm wondering if it's possible to find similar files using a shell script as well.

share|improve this question
It would probably be easier to compare one file to all other files in a directory. I'm still not sure how I'd sort the pairs of files by similarity, though. – Anderson Green Jan 8 '13 at 7:58
Actually, I found a relevant search result just now. google.com/… – Anderson Green Jan 8 '13 at 7:58
I found a program called binsort that might be able to do this, but I haven't tested it yet. neoscientists.org/~tmueller/binsort – Anderson Green Jan 8 '13 at 8:00
Alternatively, I could simply obtain the Levenshtein distance for each pair of text files. Also, a list of string metrics can be found here: en.wikipedia.org/wiki/String_metrics – Anderson Green Jan 8 '13 at 20:24
Here's a related question (about finding the longest common subsequence on Unix): stackoverflow.com/questions/9383067/… – Anderson Green Jan 8 '13 at 20:36

I think this question is too broad on so many levels. Term "difference" depends on type of data and its container: txt, mp3, avi, jpg. For each of them you need to execute their own processing methods. For instance, text or source code files may require only diff utility. Music, video and image files require fuzzy logic and computer learning algorithms.

share|improve this answer
By the way, for example geeqie image viewer has some algorithms for comparing images. – peterph Jan 8 '13 at 13:43
What I really want to do is group files by binary similarity - that's why I posted a link to binsort here. – Anderson Green Jan 8 '13 at 17:42
Also, there is a program called diff which can be used to find the difference between two files. Also, I'm mainly concerned with text and source code files here. – Anderson Green Jan 8 '13 at 17:43
Also, it would probably be easier to compare files of just one file type, instead of comparing files of all types. I think text files and source code files would probably be the easiest to compare. – Anderson Green Jan 8 '13 at 17:51
I've narrowed down the question's scope to discuss only text files and source files, so the question is no longer too broad. – Anderson Green Jan 8 '13 at 17:53

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.