I have many (i.e. >> 2) text files. These files are all mostly similar but have some small differences between them. I would like to find the lines that most commonly appear among these files. For instance, here is the type of information I am looking for (I only care about the data; by no means is any particular formatting necessary):

$ cat file1
$ cat file2
$ cat file3
$ cat file4
$ my_awesome_script file{1,2,3,4}
count | line
    3 | line1line1line1
    2 | line2line2line2
    2 | line3line3line3
    1 | line4line4line4

There are some common tools to do similar tasks, however, they have some limitations:

  • comm - can only compare two files at once; throws away lines that don't appear in both files
  • grep -f - throws away lines that don't match
  • grep -c - counts matches, but outputs a total count per file and throws away information about which input patterns match

For clarification's sake, if a line appears multiple times in one file, I only count that as one match. I just care about how many files the lines appear in, not how many times the lines appear in total.

  • Thank you @SatoKatsura. I totally forgot about uniq. Please post that as an answer so I can accept it. – jayhendren Nov 29 '16 at 18:17

Something like this should do what you want:

#! /bin/sh
for f in ./file*; do
    sort -u "$f"
done | \
    sort | \
    uniq -c | \
    sort -rn | \
    head -40

The point of the for is to make sure each line is counted only once per file. The point of the rest of the pipeline is to count the occurrences.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.