Multiple pattern match and print in single line

Question

I need to match two patterns in a log file and need to get the next line of the one of the pattern (out of two patterns) that is matched, finally need to print these three values in a single line.

Sample Log File:

2013/09/05 04:26:00          Processing Batch /fbc/dev/cebi/dod/9739867262
2013/09/05 04:26:02          Batch 9739867262 was successful
2013/09/05 04:26:02          Total Time          =  3.13 Secs
2013/09/05 04:26:02          Repository API Time =  2.96 Secs
2013/09/05 04:26:02          File System Io Time =  0.06 Secs
2013/09/05 04:26:02          Doc Validation Time =  0.03 Secs
2013/09/05 04:26:02      Ending @ Thu Sep 05 04:26:02 EDT 2013
2013/09/05 08:18:10      Starting @ Thu Sep 05 08:18:10 EDT 2013
2013/09/05 08:18:10      Starting @ Thu Sep 05 08:18:10 EDT 2013
2013/09/05 08:18:10          Processing Batch /fbc/dev/cebi/dod/9844867675
2013/09/05 08:18:10          Processing Batch /fbc/dev/cebi/dod/9886743777
2013/09/05 08:18:16          Batch 9844867675 was successful
2013/09/05 08:18:16          Total Time          =  6.00 Secs
2013/09/05 08:18:16          Repository API Time =  5.63 Secs
2013/09/05 08:18:16          File System Io Time =  0.05 Secs
2013/09/05 08:18:16          Doc Validation Time =  0.19 Secs
2013/09/05 08:18:16      Ending @ Thu Sep 05 08:18:16 EDT 2013
2013/09/05 08:18:18          Batch 9886743777 was successful
2013/09/05 08:18:18          Total Time          =  8.27 Secs
2013/09/05 08:18:18          Repository API Time =  8.52 Secs
2013/09/05 08:18:18          File System Io Time =  0.08 Secs
2013/09/05 08:18:18          Doc Validation Time =  0.47 Secs
2013/09/05 08:18:18      Ending @ Thu Sep 05 08:18:18 EDT 2013

I have the numbers seperatly in file named cust_no.txt

9739867262
9844867675
9886743777

Taking these numbers as input I need to match following two patterns in the log file

1. Processing Batch /fbc/dev/cebi/dod/
2. Batch was successful

Following are required as output:

-> On the match of first pattern (i.e Processing Batch /fbc/dev/cebi/dod/<numbers in the cust_no.txt>) I need to get the 2nd word i.e $2 . -> On the match of second patterns (i.e Batch <numbers in the cust_no.txt> was successful) I need to get the 2nd word i.e $2 -> And the 6th word ($6) on the next line after the match after the second pattern (i.e the line staring with Total Time )

Desired output:

9739867262,04:26:00,04:26:02,3.13 Secs
9844867675,08:18:10,08:18:16,6.00 Secs
9886743777,08:18:10,08:18:18,8.27 Secs

In order get this I tried in the below way but this seems not to work:

awk -v cn=$cust_no '{{if ($0 ~ "Processing.*" cn) st=$2 && if ($0 ~ "Customer cn was successful" et=$2; getline; tt=$4} ; print st,et,tt}

Answer 1

How about this:

while read number;do
    start=$(grep "Processing Batch /fbc/dev/cebi/dod/$number" log_file\
            |head -n 1|awk '{print $2}')
    end=$(grep -A 1 "Batch $number was successful" log_file\
            |head -n 2|tail -n 1|awk -v OFS=',' '{print $2,$6}')
    echo "$number,$start,$end Secs"
done <cust_no.txt

Answer 2

If you don't mind using Perl & grep here's an solution to your problem. Here's the script, called cmd.pl:

#!/usr/bin/env perl

use feature 'say';
#use Data::Dumper;

@file = `grep -f cust_no.txt -A 1 sample.log`;

my (%info, $secLineSeen, $time, $custno);

$secLineSeen = 0;
foreach my $line (@file) {
    if ($secLineSeen == 1) {
        #2013/09/05 08:18:18          Total Time          =  8.27 Secs
        (my $totTime) = ($line =~ m!\S+ \S+\s+Total Time\s+=\s+(\S+ Secs)!);
        $info{$custno}{totTime} = $totTime;
        $secLineSeen = 0;

    } elsif ($line =~ m/Processing Batch/) {
        #2013/09/05 08:18:10          Processing Batch /fbc/dev/cebi/dod/9844867675
    ($time, $custno) = ($line =~ m!\S+ (\S+)\s+Processing Batch.*/(\S+)!);
        $info{$custno}{onetwo} = $time;

  } elsif ($line =~ m/Batch.*successful/) {
        #2013/09/05 08:18:18          Batch 9886743777 was successful
        ($time, $custno) = ($line =~ m!\S+ (\S+)\s+Batch (\S+) was.*!);
        $info{$custno}{twotwo} = $time;
        $secLineSeen = 1;
    }
}

#print Dumper(\%info);

#9739867262,04:26:00,04:26:02,3.13 Secs
foreach my $key (sort keys %info) {
    say "$key,$info{$key}{onetwo},$info{$key}{twotwo},$info{$key}{totTime}";
}

Example

$ ./cmd.pl 
9739867262,04:26:00,04:26:02,3.13 Secs
9844867675,08:18:10,08:18:16,6.00 Secs
9886743777,08:18:10,08:18:18,8.27 Secs

Details

This Perl script first creates an array, @file, which contains the results of this command:

$ grep -f cust_no.txt -A 1 sample.log

This command takes the log file, sample.log, and selects all the lines that contain customer numbers from the file cust_no.txt, like so:

2013/09/05 04:26:00          Processing Batch /fbc/dev/cebi/dod/9739867262
2013/09/05 04:26:02          Batch 9739867262 was successful
2013/09/05 04:26:02          Total Time          =  3.13 Secs
--
2013/09/05 08:18:10          Processing Batch /fbc/dev/cebi/dod/9844867675
2013/09/05 08:18:10          Processing Batch /fbc/dev/cebi/dod/9886743777
2013/09/05 08:18:16          Batch 9844867675 was successful
2013/09/05 08:18:16          Total Time          =  6.00 Secs
--
2013/09/05 08:18:18          Batch 9886743777 was successful
2013/09/05 08:18:18          Total Time          =  8.27 Secs

This grep command does one extra special thing worth mentioning, mainly that it keeps one additional line after (-A 1) any match. This allows us to grab the line with the "Total Time" in it.

Once this data has been extracted, the Perl script then makes use of a multidimensional hash to store the results of the key pieces of data from this output, as per your requirements mentioned in the question.

The hash looks like this once we've completed processing the contents of @file:

$VAR1 = {
          '9739867262' => {
                            'twotwo' => '04:26:02',
                            'totTime' => '3.13 Secs',
                            'onetwo' => '04:26:00'
                          },
          '9886743777' => {
                            'twotwo' => '08:18:18',
                            'totTime' => '8.27 Secs',
                            'onetwo' => '08:18:10'
                          },
          '9844867675' => {
                            'twotwo' => '08:18:16',
                            'totTime' => '6.00 Secs',
                            'onetwo' => '08:18:10'
                          }
        };

Finally we loop through this hash and print the contents we've collected, in the format specified in the question.

Answer 3

I would try grep on this:

grep -EA 1 'pattern1|pattern2' file.log

Use option -E for extended regex and -A for number of lines to follow the match. Now to get this printed on one line, I can think of one very hackish way using sed:

grep -EA 1 'pattern1|pattern2' file.log | grep -v ^-- | sed 'N ; s+\n+|+g'

Passing the command N (read next line) to sed allows you to process two lines of input at a time. On the other hand, the command s+\n+|+g allows you to replace (with a divider character of your choise) or delete (if the substitution is empty) the newline between the two lines being processed, leaving only the new line at the end of the second line.

grep -v ^-- is necessary for me to get rid of the -- output by the first grep instance (see illustrative example below).

Line 1
Line 2
--
Line X
Line Y