0

I have multiple log files where I need to count the number of occurrences of a specific line which has a "unique ID" as a part of it.

So the log line looks like

<Some_ID> is related to <Unique_ID>

And the above values can be repeated any number of times in the logs,

I need to find a command that counts the unique number of occurrences of the above log line (with different IDs), in multiple files.

6
  • 3
    Welcome! Could you post a sample data? And have you tried something so far? – schrodigerscatcuriosity Dec 13 '19 at 18:32
  • Does the "Unique_ID" have to be after "is related to", or could it also occur (for counting) in the "Some_ID" field? – Jeff Schaller Dec 13 '19 at 18:37
  • 1
    grep -c Unique_ID * seems like a simplistic solution; how far does that get you? (Surely there's a duplicate question here somewhere?) – Jeff Schaller Dec 13 '19 at 18:38
  • Hi, thanks for responding guys! I cannot grep with Unique_ID as I wouldn't know all those values. And I need to find the occurrence of "is related to" with many possible Unique_IDs, and these lines can occur multiple times with same Unique_ID, so i want to find the unique occurrences. – Tania Dec 13 '19 at 19:04
  • 1
    So if there were two lines 345 is related to 123 and 678 is related to 123, those two lines would count as one entry, or two? – ExecutionByFork Dec 13 '19 at 19:06
0

If the text between "Some_ID" and "Unique_ID" is truly constant then the following should work (in other words, for example, there's no timestamp):

cat log1 log2 ... | grep "Some_ID" | sort | uniq -c | sort -n
1
  • Hi, I cannot grep with "Some_ID", as that value can be different for each occurrence, only the text "is related to" between the two values is constant. – Tania Dec 13 '19 at 19:12
0

This command will remove all the text prior to the unique ID, then tally the number of occurrences of unique IDs and produce a list of them, sorted in increasing order of occurrence:

cat log1 log2 ... logN | sed 's/^.* is related to //' | sort | uniq -c | sort -n
0

If I understand correctly, you want to get the IDs, we can do this with sed and poke them into a while loop to grep and wc.

cat files | sed -e 's/.*is related to //g' \
| while IFS= read -r ID; do
    count=`grep $ID files | wc -l`
    echo "$ID = $count"
done
0

From your question and your clarifications in the comments, the way I understand your problem is that you are looking for the number of unique relations. So, if given a file as follows:

345 is related to 123
678 is related to 123
187 is related to 732
678 is related to 123

The result should be 3. To do this, sort, uniq, cat, wc, and the pipe operator will be your friend.

If all the files you want to search through are in one folder, you can simply use * (a wildcard operator) to search every file. For example, cat * will print the contents of every file in the current directory. You can also specify multiple files by listing them. cat file1.txt file2.txt will print both file1 and file2. This can be combined with the wildcard to specify files in specific subdirectories: cat dir1/* dir3/* (skipping dir2/). You can also specify a partial filename: cat file* will match file1 and file123 but not afile.

Using the above, you can include any files you want into the input for sort. The pipe operator (|) takes the output of one command and pipes it into the input of another. cat file1 | sort will sort the contents of file1. You can then pipe this into uniq to filter out all the duplicate entries, and then count the lines with wc like so: cat file1 | sort | uniq | wc -l.

Using the example file I gave above the sort command will change the output into:

187 is related to 732
345 is related to 123
678 is related to 123
678 is related to 123

uniq will remove the duplicate entries (in this example at the bottom):

187 is related to 732
345 is related to 123
678 is related to 123

and wc -l will count the lines and return 3.

NOTE: This will count 123 is related to 321 and 321 is related to 123 as two separate relations. If you want to get around this you will need to do something more complicated involving parsing the input data and organizing the entries into a table or hashmap.

Also, on extremely large inputs this command chain will likely hang and look as if it has frozen. If you are using large sets of input data I would look into setting up a database to manage the data, as that will allow you to use queries to perform this type of search much faster.

2
  • 1
    Awesome! Thank you so much for explaining. I will try the above and get back to you :) – Tania Dec 13 '19 at 19:27
  • @Tania Did this work for you? If it did, please accept the answer so others can easily find this solution. – ExecutionByFork Dec 16 '19 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.