Is there an application to find similar documents?

Question

Is there any good software that finds groups of multiple files that are similar, or where one file includes a part of another?

I have a directory with lots of files mixed up. Among the files, are different versions of files, and the file names are messed up.

Answer 1

What about using diff to compare two files, counting the lines of differentiated text with 'wc -l' and then counting the actual lines in both of the compared files. If the diff lines are significantly smaller than the actual lines of the files then it's safe to say the files are more similar than not. That's a start, at least.

Here's the idea of what you'd be doing. NOTE: this is assuming you are dealing with non-binary files

First you want to get the line count of each file:

$ cat <file1> | wc -l
24
$ cat <file2> | wc -l
18
$ cat <file3> | wc -l
25
$ cat <file4> | wc -l
4

Then, go through the directory, comparing the files with 'diff' and counting the line differences:

$ diff <file1> <file2> | wc -l 
47
$ diff -ib <file1> <file3>  | wc -l
12
$ diff -ib <file1> <file4>  | wc -l
34

That's the basic idea.

The script below will do this for you (operates on current directory) and will echo out any time the diff is less than the amount of lines in either of the files. You could modify this to be more cautious, only finding matches for diff lines counts which are more than 10+ less than the line numbers of either files.

for i in `ls`; do 
  for f in `ls`; do 
    if [ $i != $f ]; then 
      F_LINES=`cat $f | wc -l`; 
      I_LINES=`cat $i | wc -l`; 
      DIFF=`diff -ib $i $f | wc -l`;
      if [ $I_LINES -ge $DIFF ]; then 
        if [ $F_LINES -ge $DIFF ]; then 
          echo "SIMILAR: $i [lines: $I_LINES] - $f [lines: $F_LINES] - # lines different: $DIFF";
        fi;
      fi;
    fi;
  done; 
done

This script, like I said, nothing special and could be greatly simplified but I'm doing this on the fly. It would output something like this (given the test file example above).

SIMILAR: <file1> [lines: 24] - <file3> [lines: 25] - # lines different: 12
SIMILAR: <file3> [lines: 25] - <file1> [lines: 24] - # lines different: 12