Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I need to process a very large log file with many lines in different formats.

My goal is to extract unique line entries who have the same starting pattern, e.g. '^2011-02-21.*MyKeyword.*Error', effectively obtaining a list of samples for each line pattern, therefore identifying the patterns.

I only know a few patterns so far, and browsing through the file manually is definitely not the option.

Please note that besides the known patterns, there is a number of unknown ones too, and I'd like to automate extracting those as well.

What is the best way to do this? I do know regular expressions quite well, but haven't done much work with awk/sed which I imagine would be used at some point in this process.

share|improve this question

2 Answers 2

If I understand correctly, you have a bunch of patterns, and you want to extract one match per pattern. The following awk script should do the trick. It prints the first occurrence of the given pattern, and records that the pattern has been seen so as not to print subsequent occurrences.

awk '
/^2011-02-21.*MyKeyword.*Error/ {
    if (!seen["^2011-02-21.*MyKeyword.*Error"]++) print;
    next;
}
1 {if (!seen[""]++) print}  # also print the first line that matches no pattern
'

Here's a variant that keeps one MyKeyword.*Error line per day.

awk '
/^[0-9]{4}-[0-9]{2}-[0-9]{2}.*MyKeyword.*Error/ {
    if (!seen[substr($0,10) "MyKeyword.*Error"]++) print;
    next;
}
'
share|improve this answer
    
Thanks Gilles, this is quite close to what I need, however I forgot to mention in my original post (re-edited it now) that I also need to identify the UNKNOWN patterns in the same fashion (one line per pattern). This would probably include some heavier script-based processing, I imagine? –  Jas Feb 21 '11 at 20:42
    
@Jas: See my edit. If you want to print all unknown lines, use 1 {print}. Note that the default handler should come last. The next keyword causes all subsequent handlers to be skipped. –  Gilles Feb 21 '11 at 20:47
    
this is certainly very helpful. Thanks. –  Jas Feb 21 '11 at 21:15

It's not clear whether you are trying to group lines by unknown output patterns or unknown keywords in known patterns.

In the first case, if you have logs such as:

[2010-04-02 12:00:00] Error: BaseController Something went wrong
2010-04-02 12:01:00 Warning - Something happened
UserController (2010-04-02 12:02:00) failed with exit status: 1
[2010-04-02 12:03:00] Error: BaseController Something went wrong
[2010-04-02 12:04:00] Error: BaseController Something went wrong
2010-04-02 12:04:01 Warning - Something else happened
UserController (2010-04-02 12:05:00) failed with exit status: 2
UserController (2010-04-02 12:06:00) failed with exit status: 10

then you could use tr and sort and/or uniq to explore the patterns:

$ tr '[:alpha:]' x < file.log | tr '[:digit:]' d | sort -u

dddd-dd-dd dd:dd:dd xxxxxxx - xxxxxxxxx xxxxxxxx
dddd-dd-dd dd:dd:dd xxxxxxx - xxxxxxxxx xxxx xxxxxxxx
[dddd-dd-dd dd:dd:dd] xxxxx: xxxxxxxxxxxxxx xxxxxxxxx xxxx xxxxx
xxxxxxxxxxxxxx (dddd-dd-dd dd:dd:dd) xxxxxx xxxx xxxx xxxxxx: d
xxxxxxxxxxxxxx (dddd-dd-dd dd:dd:dd) xxxxxx xxxx xxxx xxxxxx: dd

or, if you wanted to count occurances:

$ tr '[:alpha:]' x < file.log | tr '[:digit:]' d | sort | uniq -c
      1 
      1 dddd-dd-dd dd:dd:dd xxxxxxx - xxxxxxxxx xxxxxxxx
      1 dddd-dd-dd dd:dd:dd xxxxxxx - xxxxxxxxx xxxx xxxxxxxx
      3 [dddd-dd-dd dd:dd:dd] xxxxx: xxxxxxxxxxxxxx xxxxxxxxx xxxx xxxxx
      2 xxxxxxxxxxxxxx (dddd-dd-dd dd:dd:dd) xxxxxx xxxx xxxx xxxxxx: d
      1 xxxxxxxxxxxxxx (dddd-dd-dd dd:dd:dd) xxxxxx xxxx xxxx xxxxxx: dd

But if the pattern is always the same, such as MyKeyword is always in the same place (such as [2010-04-02 12:00:00] Error: BaseController Something went wrong) but you want to find out what strings may be in that position, you simply:

$ awk '{a[$3]++} END {for (i in a) {printf("%4d %s\n", a[i], i) } }' file.log

which should give you something like

   3 Error
   1 Info
   2 Warning
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.