Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Is there any good software that finds groups of multiple files that are similar, or where one file includes a part of another?

I have a directory with lots of files mixed up. Among the files, are different versions of files, and the file names are messed up.

share|improve this question
What format are the documents? Plain text, PDF, OpenOffice, JPEG, MP3, … – Gilles Mar 23 '12 at 23:16
You should definetly see the article about set operations in UNIX for various ways of comparing files. – user13742 Mar 24 '12 at 11:38
up vote 1 down vote accepted

What about using diff to compare two files, counting the lines of differentiated text with 'wc -l' and then counting the actual lines in both of the compared files. If the diff lines are significantly smaller than the actual lines of the files then it's safe to say the files are more similar than not. That's a start, at least.

Here's the idea of what you'd be doing. NOTE: this is assuming you are dealing with non-binary files

First you want to get the line count of each file:

$ cat <file1> | wc -l
$ cat <file2> | wc -l
$ cat <file3> | wc -l
$ cat <file4> | wc -l

Then, go through the directory, comparing the files with 'diff' and counting the line differences:

$ diff <file1> <file2> | wc -l 
$ diff -ib <file1> <file3>  | wc -l
$ diff -ib <file1> <file4>  | wc -l

That's the basic idea.

The script below will do this for you (operates on current directory) and will echo out any time the diff is less than the amount of lines in either of the files. You could modify this to be more cautious, only finding matches for diff lines counts which are more than 10+ less than the line numbers of either files.

for i in `ls`; do 
  for f in `ls`; do 
    if [ $i != $f ]; then 
      F_LINES=`cat $f | wc -l`; 
      I_LINES=`cat $i | wc -l`; 
      DIFF=`diff -ib $i $f | wc -l`;
      if [ $I_LINES -ge $DIFF ]; then 
        if [ $F_LINES -ge $DIFF ]; then 
          echo "SIMILAR: $i [lines: $I_LINES] - $f [lines: $F_LINES] - # lines different: $DIFF";

This script, like I said, nothing special and could be greatly simplified but I'm doing this on the fly. It would output something like this (given the test file example above).

SIMILAR: <file1> [lines: 24] - <file3> [lines: 25] - # lines different: 12
SIMILAR: <file3> [lines: 25] - <file1> [lines: 24] - # lines different: 12
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.