10

If I have a file example apache log file

How to extract the top most frequent error messages in a unix log file with no timestamps

the key is most frequent error message should come on top of the list

4
  • 1
    grep message logfile | sort | uniq -c | sort -n | head
    – MadHatter
    Oct 23 '13 at 5:45
  • Sorry, head should read tail.
    – MadHatter
    Oct 23 '13 at 5:58
  • Thanks for the edit, more clearly defining the question; Ursadon has answered it for you, so could you please accept his or her answer so we can stop this question popping up again? As you will gather by the slowly-accumulating downvotes, this question is arguably off-topic for SF.
    – MadHatter
    Oct 23 '13 at 7:27
  • Here there is a nice answer. I think it is suitable for many logs without any modification. They recommend to grep errors first, preparing significantly smaller logfile for analysis grep -vi -e 'info' -e 'warn'
    – Rapekas
    Jun 1 at 16:13
12

cat /tmp/file:

ERROR 1 1234
ERROR 2 1234
ERROR 3 1234
ERROR 4 1234
ERROR 4 1234
ERROR 3 1234
ERROR 2 1234
ERROR 5 1234
ERROR 1 1234
ERROR 4 1234
ERROR 1 1234
ERROR 1 1234
ERROR 1 1234
ERROR 3 1234
ERROR 2 1234
ERROR 1 1234
ERROR 4 1234
ERROR 1 1234
ERROR 4 1234
ERROR 1 1234
ERROR 2 1234

grep "ERROR" /tmp/file | sort | uniq -c | sort -r:

  8 ERROR 1 1234
  5 ERROR 4 1234
  4 ERROR 2 1234
  3 ERROR 3 1234
  1 ERROR 5 1234

first column shows how many occurrences of each string were found Explanation:

grep "ERROR" /tmp/file\ # select only ERROR string
| sort\ # order
| uniq -c\ #  count duplicate items
| sort -rn # reverse order and use numeric sort

for top 5 errors, you can add |head -n5

0
1

I think you have to chop off timestamp to get unique error

grep 'error message' /logfiles | cut -d' ' -f6- | sort | uniq -c | sort -nr

0

You might be interested in "sorting" all you system.log errors by a specific date + time too?

try: (for date only search)

grep -i "Jul 18" /var/log/*.log | sort | uniq -c | sort -n 

try: (for for a specific date + time search)

grep -i "Jul 18 16:" /var/log/*.log | sort | uniq -c | sort -n 

NOTE: for a specific date first do a cat *.log and then look for what the header says it might be different depending on the O/S. The above example is for UNIX/MacOS and you have to "manually" edit the "Jul 18 16:" with what it says in your "cat /var/log/system.log" header for your particular O/S.

Hope this helps!

:)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.