7

Should this question be moved to stackoverflow instead?

I often need to read log files generated by java applications using log4j. Usually, a logged message (let's call it a log entry) spans over multiple lines. Example:

INFO  10:57:01.123 [Thread-1] [Logger1] This is a multi-line
text, two lines
DEBUG 10:57:01.234 [Thread-1] [Logger2] This entry takes 3 lines
line 2
line 3

Note that each log entry starts at a new line and the very first word from the line is TRACE, DEBUG, INFO or ERROR and at least one space. Here, there are 2 log entry, the first at millisecond 123, the other at millisecond 234.

I would like a fast command (using a combination of sed/grep/awk/etc) to filter log entries (grep only filters lines), eg: remove all the log entries containing text 'Logger2'.

I considered doing the following transformations:

1) join lines belonging to the same log entries with a special sequence of chars (eg: ##); this way, all the log entries will take exactly one line

INFO  10:57:01.123 [Thread-1] [Logger1] This is a multi-line##text, two lines
DEBUG 10:57:01.234 [Thread-1] [Logger2] This entry takes 3 lines##line 2##line 3

2) grep
3) split the lines back (ie: replace ## with \n)

I had troubles at step 1 - I do not have enough experience with sed.

Perhaps the 3 steps above are not required, maybe sed can do all the work.

4
  • The answers at stackoverflow.com/questions/9605232/merge-two-lines-into-one may give you some ideas.
    – pmg
    Apr 18, 2015 at 9:01
  • 1
    @pmg The answear you have mentioned handles paragraphs of fixed number of lines.
    – AnFi
    Apr 18, 2015 at 9:19
  • @AndrzejA.Filip: that's the question ... at least one answer handles paragraphs with unknown number of lines
    – pmg
    Apr 18, 2015 at 9:25
  • Any reason for the down vote? Apr 18, 2015 at 9:44

3 Answers 3

5

There is no need to mix many instruments. Task can be done by sed only

sed '/^INFO\|^DEBUG\|^TRACE\|^ERROR/{
         /Logger2/{
             :1
             N
             /\nINFO\|\nDEBUG\|\nTRACE\|\nERROR/!s/\n//
             $!t1
             D     }
                                    }' log.entry
4
  • How about both positive and negative filtering? eg: include lines containing words inc1 or inc2, but exclude entries with exc1 or with exc2 Apr 18, 2015 at 9:39
  • 1
    @botismarius On default all are included. If you wants to exclude two or more use form /exc1\|exc2/ in place of /Logger2/. If you want to exclude everything exept inc1,inc2 use /inc1\|inc2/!. If you need more complex variant please show example.
    – Costas
    Apr 18, 2015 at 9:44
  • I think I can manage from here. I definitely need to learn more about sed. Would you recommend any tutorial/book in particular? Apr 18, 2015 at 11:05
  • 3
    Sed - An Introduction and Tutorial
    – Costas
    Apr 18, 2015 at 11:25
2

perl filter for multiline log records (record begin mark)

Use the following perl script as a working prototype.
Usage script_path regular_expression log_files
e.g. script_path "line \d" log_file_1 log_file_2

#!/usr/bin/perl
$pattern = qr/(?^s)$ARGV[0]/; shift; # process filtering expression
# (?^s) - treats matched string as single line
my $line = ''; # accumulates current log file record/paragraph
while(<>) {
 if( /^(TRACE|DEBUG|INFO|ERROR) /o ) { # start of new record
   &flush; # flush/print previous recors
 }
 $line.=$_;
}
&flush;
exit;

sub flush {
  local $_ = $line;
  if( length($_) and /$pattern/ ) {
    print;
  }
  $line = '';
}
1
  • This is useful. However, I preferred the answer sed as it had good chances of being faster. Apr 19, 2015 at 7:51
2

Based on one answer at https://stackoverflow.com/questions/9605232/merge-two-lines-into-one this seems to fit the bill

#!/usr/local/bin/bash

PATTERN1='TRACE *';
PATTERN2='DEBUG *';
PATTERN3='INFO *';
PATTERN4='ERROR *';
LINEOUT=""
while read line; do
    case $line in
        $PATTERN1)
                echo $LINEOUT
                LINEOUT="$line"
                        ;;
        $PATTERN2)
                echo $LINEOUT
                LINEOUT="$line"
                        ;;
        $PATTERN3)
                echo $LINEOUT
                LINEOUT="$line"
                        ;;
        $PATTERN4)
                echo $LINEOUT
                LINEOUT="$line"
                        ;;
        "")
                LINEOUT=""
                ;;

        *)      LINEOUT="$LINEOUT ## $line"
                ;;
    esac        
done
echo $LINEOUT

NB: This will add a blank space at the begining of output

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.