0

I'm writing a ksh script to parse a log file and send an email when important messages are found. Some messages are informational and I'd like to ignore them.

The log file has format

2018-01-24.08.24.35.875675    some text

    more text
    more text
    more text
    more text

2018-01-24.08.24.37.164538    some text

    more text
    more text
    INF9999W        <-- informational text
    more text

2018-01-24.08.24.46.8602545    some text

    more text
    more text
    more text

The timestamp would be considered the message separator, with the timestamp belonging to the message that follows it. I'd like to search the file for each occurrences of "information text" and then remove that whole message from the file (from preceding timestamp until just before the next timestamp).

How can I easily determine the line numbers of the preceding and following timestamps so I remove those lines with:

awk 'NR<'$preceding_ts' || NR >='$following_ts'

My way is to get all the timestamp lines into a file and then loop through that file until I find the timestamp lines that are just before and after the 'information-text' line #. It seems like a lot of work, particularly on a large file. Is there a more efficient way.

integer inf_line
integer last_ts_line
integer cur_ts
cp $error_log $copy_log
while true
do
   inf_line=$(grep -n "INF99999W" $copy_log | head -1 | cut -f1 -d":")
   if [[ $inf_line -eq 0 ]]
   then
      break
   fi     
   grep -n -E "^20[0-9][0-9]-[0-1][0-9]-[0-3][0-9]-" $copy_log | cut -f1 -d":" > $ts_lines
   last_ts_line=99999999
   cat $ts_lines | while read cur_ts
   do       
      if [[ $cur_ts -gt $inf_line && $last_ts_line -lt $inf_line ]]
      then
         awk 'NR<'$last_ts_line' || NR >='$cur_ts'' $copy_log > $temp_log
         cp $temp_log $copy_log
         last_ts_line=$cur_ts
         break
      fi
      last_ts_line=$cur_ts
   done
   if [[ $last_ts_line -lt $inf_line ]]
   then
      awk 'NR<'$last_ts_line'' $copy_log > $temp_log
      cp $temp_log $copy_log
   fi
done

Thanks.

2

I would approach it by storing the lines of the current message, and when the message ends, printing the stored lot if there was no INF marker seen. Here, d holds the lines of the current message (d for data), p tells if we want to print the stored lines or not.

awk -vinfo='INF99+' \
    '/^20[0-9][0-9]-[0-1][0-9]-[0-3][0-9]/ {
         if (p) printf "%s", d; d = $0 ORS; p=1; next } 
     $0 ~ info {p=0} 
     {d = d $0 ORS} 
     END {if (p) printf "%s", d}' < log 

The first rule here matches on the timestamp lines, prints any stored lines if p is true, stores this line and sets p to one. The second rules resets p to zero if a line with the info pattern is seen; the pattern is set to the variable with -vinfo=.... The third rule appends the current line to those collected, and the END rule again just prints the collected lines if p is set.


We could also write it like this, this would check the info pattern also on the timestamp line:

awk -vinfo='INF99+' \
    '/^20[0-9][0-9]-[0-1][0-9]-[0-3][0-9]/ {
         if (p) { printf "%s", d }; d = ""; p=1; } 
     $0 ~ info {p=0} 
     {d = d $0 ORS} 
     END {if (p) printf "%s", d}' < log 

In general, it's probably a good idea to write stuff like this in awk or Perl. The result will at least be much faster to run than a shell script that forks dozens of copies of grep, awk, and cut etc...

  • Wow. I need to brush up on my awk skills. I've typically used it for simple operations. Your code works as advertised. I'll be using a file to supply a list of "INF99" type values. How do I use a variable in your expression to pass in those values? Replacing your INF99 with $string didn't do it (again, my sub-par awk skills). Thanks. – Scavenger Jan 24 '18 at 18:10
  • @Scavenger, it's easier to pass stuff like that as variables to awk with -vvar=value (or -vvar="$shellvar"), see edit. The script is in single quotes, so the shell will not expand variables inside it, and if you change it to double quotes, then you'll need to escape all the other $ signs inside it. – ilkkachu Jan 24 '18 at 18:24
  • Just trying to wrap my head around this. Would it make sense to change {p=0} to {p=0; next} to avoid adding to d since it will be discarded anyway? – Scavenger Jan 24 '18 at 20:33
  • @Scavenger, you could do that, or you could put p as a condition for adding to d ( p {d = d $0 ORS} ), but I doubt it makes much difference (and in any case, the script might collect any number of lines before seeing that the current block should be dropped.) – ilkkachu Jan 24 '18 at 20:38
  • If there's a chance that the timestamp line could also have the text to exclude (not in my initial example), would I just remove the 'next' after p=1 so the line gets excluded by the "$0~ info {p=0}" line? Why does the 2nd rule not need braces around it like the 3rd rule does? Thanks again. Fascinating stuff. – Scavenger Jan 24 '18 at 20:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.