What about using diff to compare two files, counting the lines of differentiated text with 'wc -l' and then counting the actual lines in both of the compared files. If the diff lines are significantly smaller than the actual lines of the files then it's safe to say the files are more similar than not. That's a start, at least.
Here's the idea of what you'd be doing. NOTE: this is assuming you are dealing with non-binary files
First you want to get the line count of each file:
$ cat <file1> | wc -l
$ cat <file2> | wc -l
$ cat <file3> | wc -l
$ cat <file4> | wc -l
Then, go through the directory, comparing the files with 'diff' and counting the line differences:
$ diff <file1> <file2> | wc -l
$ diff -ib <file1> <file3> | wc -l
$ diff -ib <file1> <file4> | wc -l
That's the basic idea.
The script below will do this for you (operates on current directory) and will echo out any time the diff is less than the amount of lines in either of the files. You could modify this to be more cautious, only finding matches for diff lines counts which are more than 10+ less than the line numbers of either files.
for i in `ls`; do
for f in `ls`; do
if [ $i != $f ]; then
F_LINES=`cat $f | wc -l`;
I_LINES=`cat $i | wc -l`;
DIFF=`diff -ib $i $f | wc -l`;
if [ $I_LINES -ge $DIFF ]; then
if [ $F_LINES -ge $DIFF ]; then
echo "SIMILAR: $i [lines: $I_LINES] - $f [lines: $F_LINES] - # lines different: $DIFF";
This script, like I said, nothing special and could be greatly simplified but I'm doing this on the fly. It would output something like this (given the test file example above).
SIMILAR: <file1> [lines: 24] - <file3> [lines: 25] - # lines different: 12
SIMILAR: <file3> [lines: 25] - <file1> [lines: 24] - # lines different: 12