I have many (i.e. >> 2) text files. These files are all mostly similar but have some small differences between them. I would like to find the lines that most commonly appear among these files. For instance, here is the type of information I am looking for (I only care about the data; by no means is any particular formatting necessary):

$ cat file1
line1line1line1
line2line2line2
line3line3line3
$ cat file2
line1line1line1
line2line2line2
$ cat file3
line1line1line1
line3line3line3
$ cat file4
line4line4line4
$ my_awesome_script file{1,2,3,4}
count | line
------+----------------
    3 | line1line1line1
    2 | line2line2line2
    2 | line3line3line3
    1 | line4line4line4

There are some common tools to do similar tasks, however, they have some limitations:

  • comm - can only compare two files at once; throws away lines that don't appear in both files
  • grep -f - throws away lines that don't match
  • grep -c - counts matches, but outputs a total count per file and throws away information about which input patterns match

For clarification's sake, if a line appears multiple times in one file, I only count that as one match. I just care about how many files the lines appear in, not how many times the lines appear in total.

  • Thank you @SatoKatsura. I totally forgot about uniq. Please post that as an answer so I can accept it. – jayhendren Nov 29 '16 at 18:17
up vote 3 down vote accepted

Something like this should do what you want:

#! /bin/sh
for f in ./file*; do
    sort -u "$f"
done | \
    sort | \
    uniq -c | \
    sort -rn | \
    head -40

The point of the for is to make sure each line is counted only once per file. The point of the rest of the pipeline is to count the occurrences.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.