5

Should this question be moved to stackoverflow instead?

I often need to read log files generated by java applications using log4j. Usually, a logged message (let's call it a log entry) spans over multiple lines. Example:

INFO  10:57:01.123 [Thread-1] [Logger1] This is a multi-line
text, two lines
DEBUG 10:57:01.234 [Thread-1] [Logger2] This entry takes 3 lines
line 2
line 3

Note that each log entry starts at a new line and the very first word from the line is TRACE, DEBUG, INFO or ERROR and at least one space. Here, there are 2 log entry, the first at millisecond 123, the other at millisecond 234.

I would like a fast command (using a combination of sed/grep/awk/etc) to filter log entries (grep only filters lines), eg: remove all the log entries containing text 'Logger2'.

I considered doing the following transformations:

1) join lines belonging to the same log entries with a special sequence of chars (eg: ##); this way, all the log entries will take exactly one line

INFO  10:57:01.123 [Thread-1] [Logger1] This is a multi-line##text, two lines
DEBUG 10:57:01.234 [Thread-1] [Logger2] This entry takes 3 lines##line 2##line 3

2) grep
3) split the lines back (ie: replace ## with \n)

I had troubles at step 1 - I do not have enough experience with sed.

Perhaps the 3 steps above are not required, maybe sed can do all the work.

  • The answers at stackoverflow.com/questions/9605232/merge-two-lines-into-one may give you some ideas. – pmg Apr 18 '15 at 9:01
  • 1
    @pmg The answear you have mentioned handles paragraphs of fixed number of lines. – AnFi Apr 18 '15 at 9:19
  • @AndrzejA.Filip: that's the question ... at least one answer handles paragraphs with unknown number of lines – pmg Apr 18 '15 at 9:25
  • Any reason for the down vote? – botismarius Apr 18 '15 at 9:44
5

There is no need to mix many instruments. Task can be done by sed only

sed '/^INFO\|^DEBUG\|^TRACE\|^ERROR/{
         /Logger2/{
             :1
             N
             /\nINFO\|\nDEBUG\|\nTRACE\|\nERROR/!s/\n//
             $!t1
             D     }
                                    }' log.entry
  • How about both positive and negative filtering? eg: include lines containing words inc1 or inc2, but exclude entries with exc1 or with exc2 – botismarius Apr 18 '15 at 9:39
  • 1
    @botismarius On default all are included. If you wants to exclude two or more use form /exc1\|exc2/ in place of /Logger2/. If you want to exclude everything exept inc1,inc2 use /inc1\|inc2/!. If you need more complex variant please show example. – Costas Apr 18 '15 at 9:44
  • I think I can manage from here. I definitely need to learn more about sed. Would you recommend any tutorial/book in particular? – botismarius Apr 18 '15 at 11:05
  • 3
    Sed - An Introduction and Tutorial – Costas Apr 18 '15 at 11:25
2

perl filter for multiline log records (record begin mark)

Use the following perl script as a working prototype.
Usage script_path regular_expression log_files
e.g. script_path "line \d" log_file_1 log_file_2

#!/usr/bin/perl
$pattern = qr/(?^s)$ARGV[0]/; shift; # process filtering expression
# (?^s) - treats matched string as single line
my $line = ''; # accumulates current log file record/paragraph
while(<>) {
 if( /^(TRACE|DEBUG|INFO|ERROR) /o ) { # start of new record
   &flush; # flush/print previous recors
 }
 $line.=$_;
}
&flush;
exit;

sub flush {
  local $_ = $line;
  if( length($_) and /$pattern/ ) {
    print;
  }
  $line = '';
}
  • This is useful. However, I preferred the answer sed as it had good chances of being faster. – botismarius Apr 19 '15 at 7:51
2

Based on one answer at https://stackoverflow.com/questions/9605232/merge-two-lines-into-one this seems to fit the bill

#!/usr/local/bin/bash

PATTERN1='TRACE *';
PATTERN2='DEBUG *';
PATTERN3='INFO *';
PATTERN4='ERROR *';
LINEOUT=""
while read line; do
    case $line in
        $PATTERN1)
                echo $LINEOUT
                LINEOUT="$line"
                        ;;
        $PATTERN2)
                echo $LINEOUT
                LINEOUT="$line"
                        ;;
        $PATTERN3)
                echo $LINEOUT
                LINEOUT="$line"
                        ;;
        $PATTERN4)
                echo $LINEOUT
                LINEOUT="$line"
                        ;;
        "")
                LINEOUT=""
                ;;

        *)      LINEOUT="$LINEOUT ## $line"
                ;;
    esac        
done
echo $LINEOUT

NB: This will add a blank space at the begining of output

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.