Count number of unique occurrences in multiple files

Question

I have multiple log files where I need to count the number of occurrences of a specific line which has a "unique ID" as a part of it.

So the log line looks like

<Some_ID> is related to <Unique_ID>

And the above values can be repeated any number of times in the logs,

I need to find a command that counts the unique number of occurrences of the above log line (with different IDs), in multiple files.

Answer 1

If the text between "Some_ID" and "Unique_ID" is truly constant then the following should work (in other words, for example, there's no timestamp):

cat log1 log2 ... | grep "Some_ID" | sort | uniq -c | sort -n

Answer 2

This command will remove all the text prior to the unique ID, then tally the number of occurrences of unique IDs and produce a list of them, sorted in increasing order of occurrence:

cat log1 log2 ... logN | sed 's/^.* is related to //' | sort | uniq -c | sort -n

Answer 3

If I understand correctly, you want to get the IDs, we can do this with sed and poke them into a while loop to grep and wc.

cat files | sed -e 's/.*is related to //g' \
| while IFS= read -r ID; do
    count=`grep $ID files | wc -l`
    echo "$ID = $count"
done

Answer 4

From your question and your clarifications in the comments, the way I understand your problem is that you are looking for the number of unique relations. So, if given a file as follows:

345 is related to 123
678 is related to 123
187 is related to 732
678 is related to 123

The result should be 3. To do this, sort, uniq, cat, wc, and the pipe operator will be your friend.

If all the files you want to search through are in one folder, you can simply use * (a wildcard operator) to search every file. For example, cat * will print the contents of every file in the current directory. You can also specify multiple files by listing them. cat file1.txt file2.txt will print both file1 and file2. This can be combined with the wildcard to specify files in specific subdirectories: cat dir1/* dir3/* (skipping dir2/). You can also specify a partial filename: cat file* will match file1 and file123 but not afile.

Using the above, you can include any files you want into the input for sort. The pipe operator (|) takes the output of one command and pipes it into the input of another. cat file1 | sort will sort the contents of file1. You can then pipe this into uniq to filter out all the duplicate entries, and then count the lines with wc like so: cat file1 | sort | uniq | wc -l.

Using the example file I gave above the sort command will change the output into:

187 is related to 732
345 is related to 123
678 is related to 123
678 is related to 123

uniq will remove the duplicate entries (in this example at the bottom):

187 is related to 732
345 is related to 123
678 is related to 123

and wc -l will count the lines and return 3.

NOTE: This will count 123 is related to 321 and 321 is related to 123 as two separate relations. If you want to get around this you will need to do something more complicated involving parsing the input data and organizing the entries into a table or hashmap.

Also, on extremely large inputs this command chain will likely hang and look as if it has frozen. If you are using large sets of input data I would look into setting up a database to manage the data, as that will allow you to use queries to perform this type of search much faster.