1

Is there a way in Unix, to evaluate how similar or dissimilar, two text files are?

I know about diff, but it just gives me the differences itself. What I am after is to just receive as an answer, some kind of evaluation of how much alike those two texts are. Do they contain much the same words, are the texts often quite similar etc. It is actually quite complicated to evaluate this in a high quality manner so I'm wondering if somebody has already done this.

Would be good to just receive a number, could then be that 0 means they are identical, and a high number means they are nothing alike.

|improve this question
  • 4
    There's an active research field related to this. Depending on what you want to do, answers can range from n-grams to sentiment analysis, to opinion mining, to automatic corpora classification. – Satō Katsura Nov 1 '16 at 9:55
  • Do you know of any open source solutions? Would be a shame if I spent work on something thats already been done and open sourced somewhere else. – Hermann Ingjaldsson Nov 1 '16 at 10:43
  • Your question is too broad to have an useful answer. – Satō Katsura Nov 1 '16 at 10:49
  • How could I narrow it down? – Hermann Ingjaldsson Nov 1 '16 at 10:53
2

There is diffstat which reads normal diff output and prints some stats more about lines only.

diff -u fileA fileB | diffstat

or wdiff to handle word diffs.

wdiff -123 --statistics fileA fileB

You may also read the man pages to find some more interesting options.

|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.