4

I need to process a very large log file with many lines in different formats.

My goal is to extract unique line entries who have the same starting pattern, e.g. '^2011-02-21.*MyKeyword.*Error', effectively obtaining a list of samples for each line pattern, therefore identifying the patterns.

I only know a few patterns so far, and browsing through the file manually is definitely not the option.

Please note that besides the known patterns, there is a number of unknown ones too, and I'd like to automate extracting those as well.

What is the best way to do this? I do know regular expressions quite well, but haven't done much work with awk/sed which I imagine would be used at some point in this process.

Improve this question

2 Answers 2

Reset to default
2

If I understand correctly, you have a bunch of patterns, and you want to extract one match per pattern. The following awk script should do the trick. It prints the first occurrence of the given pattern, and records that the pattern has been seen so as not to print subsequent occurrences.

awk '
/^2011-02-21.*MyKeyword.*Error/ {
    if (!seen["^2011-02-21.*MyKeyword.*Error"]++) print;
    next;
}
1 {if (!seen[""]++) print}  # also print the first line that matches no pattern
'

Here's a variant that keeps one MyKeyword.*Error line per day.

awk '
/^[0-9]{4}-[0-9]{2}-[0-9]{2}.*MyKeyword.*Error/ {
    if (!seen[substr($0,10) "MyKeyword.*Error"]++) print;
    next;
}
'
Improve this answer
2
  • Thanks Gilles, this is quite close to what I need, however I forgot to mention in my original post (re-edited it now) that I also need to identify the UNKNOWN patterns in the same fashion (one line per pattern). This would probably include some heavier script-based processing, I imagine?
    – Jas
    Feb 21, 2011 at 20:42
  • @Jas: See my edit. If you want to print all unknown lines, use 1 {print}. Note that the default handler should come last. The next keyword causes all subsequent handlers to be skipped.
    – Gilles 'SO- stop being evil'
    Feb 21, 2011 at 20:47
1

It's not clear whether you are trying to group lines by unknown output patterns or unknown keywords in known patterns.

In the first case, if you have logs such as:

[2010-04-02 12:00:00] Error: BaseController Something went wrong
2010-04-02 12:01:00 Warning - Something happened
UserController (2010-04-02 12:02:00) failed with exit status: 1
[2010-04-02 12:03:00] Error: BaseController Something went wrong
[2010-04-02 12:04:00] Error: BaseController Something went wrong
2010-04-02 12:04:01 Warning - Something else happened
UserController (2010-04-02 12:05:00) failed with exit status: 2
UserController (2010-04-02 12:06:00) failed with exit status: 10

then you could use tr and sort and/or uniq to explore the patterns:

$ tr '[:alpha:]' x < file.log | tr '[:digit:]' d | sort -u

dddd-dd-dd dd:dd:dd xxxxxxx - xxxxxxxxx xxxxxxxx
dddd-dd-dd dd:dd:dd xxxxxxx - xxxxxxxxx xxxx xxxxxxxx
[dddd-dd-dd dd:dd:dd] xxxxx: xxxxxxxxxxxxxx xxxxxxxxx xxxx xxxxx
xxxxxxxxxxxxxx (dddd-dd-dd dd:dd:dd) xxxxxx xxxx xxxx xxxxxx: d
xxxxxxxxxxxxxx (dddd-dd-dd dd:dd:dd) xxxxxx xxxx xxxx xxxxxx: dd

or, if you wanted to count occurances:

$ tr '[:alpha:]' x < file.log | tr '[:digit:]' d | sort | uniq -c
      1 
      1 dddd-dd-dd dd:dd:dd xxxxxxx - xxxxxxxxx xxxxxxxx
      1 dddd-dd-dd dd:dd:dd xxxxxxx - xxxxxxxxx xxxx xxxxxxxx
      3 [dddd-dd-dd dd:dd:dd] xxxxx: xxxxxxxxxxxxxx xxxxxxxxx xxxx xxxxx
      2 xxxxxxxxxxxxxx (dddd-dd-dd dd:dd:dd) xxxxxx xxxx xxxx xxxxxx: d
      1 xxxxxxxxxxxxxx (dddd-dd-dd dd:dd:dd) xxxxxx xxxx xxxx xxxxxx: dd

But if the pattern is always the same, such as MyKeyword is always in the same place (such as [2010-04-02 12:00:00] Error: BaseController Something went wrong) but you want to find out what strings may be in that position, you simply:

$ awk '{a[$3]++} END {for (i in a) {printf("%4d %s\n", a[i], i) } }' file.log

which should give you something like

   3 Error
   1 Info
   2 Warning
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .