I got this project to analysis my logs where I need to find any consecutive errors occurred more then thrice and list logs name. As my logs were huge, I wrote below script:

for i in `ls`  
  count=`uniq -c $i | grep 'FATAL ERROR' |sed 's/^ *//g'| sed 's/ /,/g' | awk -F',' '{if($1 >1) {print $1}}'`  
  if [[ $count -ge 2 ]]  
    echo $i  

This works if you just need to know number of occurrence in which all file

grep -o -c Source * | awk -F: '{if ($2 > 2){print $1}}'  

What could have been the better version of my script? What if log is really huge, my piece of code will slowdown.

  • 2
  • @RoVo, Thanks a lot for your suggestion . – Machine Mar 26 at 9:38
  • If you're using -ge 2 you'll trigger on two or more. Your question uses "thrice" which means three times. Which is correct? – roaima Apr 7 at 16:38

Pipelines should generally be kept short, and most of the things that you do to get the value for count could be done in a single awk program, including the loop and the if statement.

awk 'FNR == 1 || !/FATAL ERROR/ { count = 0 }
     /FATAL ERROR/              { ++count }
     count == 2                 { print FILENAME }' ./*

This runs a single awk program across all non-hidden files in the current directory.

The program will reset its count variable to zero if we're at the first line of a new file or if the current line does not match the pattern FATAL ERROR.

If the line matches the pattern FATAL ERROR, the count variable is incremented.

If the count variable reaches a value of 2, the name of the current file is printed.

The code will print the name of the file each time it finds two consecutive lines that matches the pattern, even if this happens multiple times in the same file. If this in not wanted, you could expand the code slightly:

awk 'FNR == 1       { count = 0; do_print = 1 }
     !/FATAL ERROR/ { count = 0 }
     /FATAL ERROR/  { ++count }
     count == 2 && do_print { print FILENAME; do_print = 0 }' ./*

or with GNU awk (using nextfile to skip to the next file):

awk 'FNR == 1 || !/FATAL ERROR/ { count = 0 }
     /FATAL ERROR/              { ++count }
     count == 2 && do_print     { print FILENAME; nextfile }' ./*

Related to your shell loop:

  • Finding it hard to understand , but still trying :) – Machine Mar 26 at 9:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.