8

Should this question be moved to stackoverflow instead?

I often need to read log files generated by java applications using log4j. Usually, a logged message (let's call it a log entry) spans over multiple lines. Example:

INFO  10:57:01.123 [Thread-1] [Logger1] This is a multi-line
text, two lines
DEBUG 10:57:01.234 [Thread-1] [Logger2] This entry takes 3 lines
line 2
line 3

Note that each log entry starts at a new line and the very first word from the line is TRACE, DEBUG, INFO or ERROR and at least one space. Here, there are 2 log entry, the first at millisecond 123, the other at millisecond 234.

I would like a fast command (using a combination of sed/grep/awk/etc) to filter log entries (grep only filters lines), eg: remove all the log entries containing text 'Logger2'.

I considered doing the following transformations:

1) join lines belonging to the same log entries with a special sequence of chars (eg: ##); this way, all the log entries will take exactly one line

INFO  10:57:01.123 [Thread-1] [Logger1] This is a multi-line##text, two lines
DEBUG 10:57:01.234 [Thread-1] [Logger2] This entry takes 3 lines##line 2##line 3

2) grep
3) split the lines back (ie: replace ## with \n)

I had troubles at step 1 - I do not have enough experience with sed.

Perhaps the 3 steps above are not required, maybe sed can do all the work.

Improve this question
4
  • The answers at stackoverflow.com/questions/9605232/merge-two-lines-into-one may give you some ideas.
    – pmg
    Apr 18, 2015 at 9:01
  • 1
    @pmg The answear you have mentioned handles paragraphs of fixed number of lines.
    – AnFi
    Apr 18, 2015 at 9:19
  • @AndrzejA.Filip: that's the question ... at least one answer handles paragraphs with unknown number of lines
    – pmg
    Apr 18, 2015 at 9:25
  • Any reason for the down vote?
    – botismarius
    Apr 18, 2015 at 9:44

3 Answers 3

Reset to default
6

There is no need to mix many instruments. Task can be done by sed only 

sed '/^INFO\|^DEBUG\|^TRACE\|^ERROR/{
         /Logger2/{
             :1
             N
             /\nINFO\|\nDEBUG\|\nTRACE\|\nERROR/!s/\n//
             $!t1
             D     }
                                    }' log.entry
Improve this answer
4
  • How about both positive and negative filtering? eg: include lines containing words inc1 or inc2, but exclude entries with exc1 or with exc2
    – botismarius
    Apr 18, 2015 at 9:39
  • 1
    @botismarius On default all are included. If you wants to exclude two or more use form /exc1\|exc2/ in place of /Logger2/. If you want to exclude everything exept inc1,inc2 use /inc1\|inc2/!. If you need more complex variant please show example.
    – Costas
    Apr 18, 2015 at 9:44
  • I think I can manage from here. I definitely need to learn more about sed. Would you recommend any tutorial/book in particular?
    – botismarius
    Apr 18, 2015 at 11:05
  • 3
    Sed - An Introduction and Tutorial
    – Costas
    Apr 18, 2015 at 11:25
2

perl filter for multiline log records (record begin mark)

Use the following perl script as a working prototype.
Usage script_path regular_expression log_files
e.g. script_path "line \d" log_file_1 log_file_2

#!/usr/bin/perl
$pattern = qr/(?^s)$ARGV[0]/; shift; # process filtering expression
# (?^s) - treats matched string as single line
my $line = ''; # accumulates current log file record/paragraph
while(<>) {
 if( /^(TRACE|DEBUG|INFO|ERROR) /o ) { # start of new record
   &flush; # flush/print previous recors
 }
 $line.=$_;
}
&flush;
exit;

sub flush {
  local $_ = $line;
  if( length($_) and /$pattern/ ) {
    print;
  }
  $line = '';
}
Improve this answer
1
  • This is useful. However, I preferred the answer sed as it had good chances of being faster.
    – botismarius
    Apr 19, 2015 at 7:51
2

Based on one answer at https://stackoverflow.com/questions/9605232/merge-two-lines-into-one this seems to fit the bill

#!/usr/local/bin/bash

PATTERN1='TRACE *';
PATTERN2='DEBUG *';
PATTERN3='INFO *';
PATTERN4='ERROR *';
LINEOUT=""
while read line; do
    case $line in
        $PATTERN1)
                echo $LINEOUT
                LINEOUT="$line"
                        ;;
        $PATTERN2)
                echo $LINEOUT
                LINEOUT="$line"
                        ;;
        $PATTERN3)
                echo $LINEOUT
                LINEOUT="$line"
                        ;;
        $PATTERN4)
                echo $LINEOUT
                LINEOUT="$line"
                        ;;
        "")
                LINEOUT=""
                ;;

        *)      LINEOUT="$LINEOUT ## $line"
                ;;
    esac        
done
echo $LINEOUT

NB: This will add a blank space at the begining of output

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .