Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have very big logs (several gigabytes per day), that can (but do not need to) contain specific lines. I have to count the number of occurences of every one of these lines on a daily basis.

I have a file patterns.in, that contains the desired lines. For example:

aaaa
bbbb
cccc
dddd
eeee
ffff

The log files can look like this:

asd
dfg
aaaa
aaaa
sa
sdf
dddd
dddd
dddd
dddd
ghj
bbbb
cccc
cccc
cccc
fgg
fgh
hjk

The first (and perhaps most obvious approach) is to use grep, sort and uniq in the following way:

grep -f patterns.in logfile.txt | sort | uniq -c

which gives the following result:

   2 aaaa
   1 bbbb
   3 cccc
   4 dddd

It is close to what I want to achieve, but my desired result is:

   2 aaaa
   1 bbbb
   3 cccc
   4 dddd
   0 eeee
   0 ffff

So the problem is: how to print '0' if a line from pattern.in file is not matched? It needs to be done in a simplest possible way, as all I have available is the cygwin environment.

share|improve this question
1  
BTW, you might need grep's -o option otherwise multiple lines that, e.g., contain 'aaaa' plus other (different) data will be treated as non-unique by the sort | uniq -c –  cas Oct 11 '12 at 5:36
    
Well spotted, thank you! –  gorkypl Oct 11 '12 at 9:51
add comment

1 Answer

up vote 7 down vote accepted

how about feeding the pattern file back in as a data file so that each pattern finds at least one match, and then subtracting one from the final reported count for each match

grep -f patterns.in logfile.txt patterns.in | cut -f2 -d':' | sort | uniq -c | awk '{print($1 - 1" "$2)}'
share|improve this answer
    
Seems to be working well on examples, will need to check tommorow in production, as I am not sure if I have awk available there. –  gorkypl Oct 10 '12 at 18:33
2  
+1, nice answer. you can use grep's -h or --no-filename option to stop grep from printing filenames. e.g. grep -h -o -f patterns.in logfile.txt patterns.in sort | uniq -c | awk '{print($1 - 1" "$2)}' –  cas Oct 11 '12 at 5:37
    
It almost works, with one exception - awk '{print($1 - 1" "$2)}' prints only the first word of matched line (second field, $2). If a line has multiple words, how can I write its whole contents, that is from $2 to end of line? –  gorkypl Oct 11 '12 at 9:52
2  
@gorkypl, incorporated Craig Sander's grep -h suggestion and modified the command to work with multiple input words, here goes. grep -h -f patterns.in logfile.txt patterns.in | sort | uniq -c | tr -s ' ' |awk ' {count=$1 - 1; file_name=$0; sub($1, "", file_name);print(count" "file_name)}' –  1_CR Oct 11 '12 at 16:57
    
Yay, great - thank you :) –  gorkypl Oct 12 '12 at 11:21
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.