1

I was breaking my head for a day still not successful.

I have this script:

#!/bin/ksh

fname=$1

for batchname in $(grep -i "Processing batch" $fname | cut -d "'" -f2)
do
Batch_state=`grep -c -i "Batch '$batchname' was successful" $fname`
if [[ "$Batch_state"  -ge 1 ]];then
{
S_Time=`awk '/[0-9]_[0-9].*successful/{getline;getline;getline;print}' $fname | awk '{print $2}'`
E_Time=`awk '/[0-9]_[0-9].*successful/{getline;getline;getline;getline;print}' $fname | awk '{print $2}'`
echo -e $batchname"\t"$S_Time"\t"$E_Time
}
else
{
echo $batchname encountered an error
}
fi
done

Output this code is producing:

02_1231324      14:29:04 15:29:11   14:32:19 15:33:11
79_3097935      14:29:04 15:29:11   14:32:19 15:33:11

Desired Output:

02_1231324      14:29:04    14:32:19
79_3097935      15:29:11    15:33:11 

Sample Input:

2013/06/11 14:29:04 <0999> (725102)
Creating batch '02_1231324.0'...
2013/06/11 14:29:04 <0999> (725102)
Batch '02_1231324' was successful
2013/06/11 14:29:04 <0999> (725102)
TMR:Child ZERO, 160 Docs 320 Pgs 3874 KByts Tot 0.42 WAL 0.10 WALIO 0.15 IO 0.03 secs
2013/06/11 14:29:04 <0999> (725102)  Processing batch '02_1231324'
2013/06/11 14:32:19 <0999> (725102)
Total in batch: 160 documents using 4 KBytes
2013/06/11 15:29:11 <0999> (725102)
Creating batch '79_3097935.0'...
2013/06/11 15:29:11 <0999> (725102)
Batch '79_3097935' was successful
2013/06/11 15:29:11 <0999> (725102)
TMR:Child ZERO, 160 Docs 320 Pgs 3874 KByts Tot 0.42 WAL 0.10 WALIO 0.15 IO 0.03 secs
2013/06/11 15:29:11 <0999> (725102)  Processing batch '79_3097935'
2013/06/11 15:33:11 <0999> (725102)
Total in batch: 160 documents using 4 KBytes
TMR:Child ZERO, 160 Docs 320 Pgs 3874 KByts Tot 0.42 WAL 0.10 WALIO 0.15 IO 0.03 secs
2013/06/11 13:26:57 <0999> (725102)  Processing batch '12_2013162201'
2013/06/11 13:26:57 <0999> (725102)
Total in batch: 160 documents using 4 KBytes

So, what is wrong with my script? How can I get the desired output?

17
  • echo ${x##* } - would print last one.
    – jirib
    Commented Oct 11, 2013 at 13:15
  • 5
    I don't understand what you're doing. Could you include the actual lines from your script? What kind of loop? You can't assign a variable that way, the value of $x will always be awk and the script won't run in what you've posted. I assume you have a $() around it?
    – terdon
    Commented Oct 11, 2013 at 13:19
  • 1
    Could you also show us an example of he input file? Basically, when you ask questions like this, you should give us everything we need to repeat what you're doing. We can't help otherwise. So always think to provide whatever people would need to get the same behavior that you're experiencing.
    – terdon
    Commented Oct 11, 2013 at 14:08
  • 2
    Thanks, but could you post the input that actually contains the desired output?Or show us what output you would want to get from the input you posted? The lines you show are not contained in your input file. You also have syntax errors in your script. Why do you have } in there?
    – terdon
    Commented Oct 11, 2013 at 14:44
  • 2
    Temporary workaround would be taking the current output (as is in your answer) and removing the bits you don't like either with sed or awk. However I have a feeling that your main script is rather messy - for example having awk ... file | awk ... twice and then just printing its output looks rather not optimal. Plus the indenting... please. Also you might want to remove pieces from the question that are no longer up-to-date.
    – peterph
    Commented Oct 11, 2013 at 14:58

4 Answers 4

3

You probably want something like this to parse your log:

#!/bin/awk -f
{
    if (/was successful/) {
        bn = $2;
        gsub(/'/, "", bn);
        succ[bn] = 1;
    }
    if (/Processing batch/) {
        bn = $7;
        gsub(/'/, "", bn);
        if (bn in succ) {
            succ[bn,",s"] = $2;
            getline
            succ[bn,",e"] = $2;
        } else
            fail[bn] = 1;
    }
}
END {
    for (val in succ)
        if (val !~ /,/)
            print val " " succ[val,",s"] " " succ[val,",e"];
    for (val in fail)
        print val " encountered an error";
}

Your original script slightly reduced to just 1 awk call under some strict1 assumptions about the structure of your input data).

#!/bin/ksh

fname=$1

for batchname in $(grep -i "Processing batch" $fname | cut -d "'" -f2); do
    Batch_state=`grep -c -i "Batch '$batchname' was successful" $fname`
    if [[ "$Batch_state"  -ge 1 ]]; then
        awk -v b=$batchname '
{
    if ($0 ~ "Processing.*" b) {
        s = $2;
        getline;
        e = $2;
        print b " " s " " e;
        exit
    }
}' $fname
    else
        echo $batchname encountered an error
    fi
done

Shorter version using grep -A (-A is a GNU extension which selects the matching line and a number - 1 by default - of lines that follow):

#!/bin/ksh

fname=$1

for batchname in $(grep -i "Processing batch" $fname | cut -d "'" -f2); do
    Batch_state=`grep -c -i "Batch '$batchname' was successful" $fname`
    printf "%s" $batchname
    if [[ "$Batch_state"  -ge 1 ]]; then
        grep -A1 "Processing batch '$batchname'" $fname | cut -f2 -d" " | fmt
    else
        printf "encountered an error\n"
    fi
done

But these are still processing the file 3 times. See ChuckCottrill's answer for how it could be done.


1 it seems that the batches are not processed in parallel, and what you want is actually the line containing Processing batch and the line after that.

2
  • :Your awk worked perfectly... Thanks a lot...:) But when tried second script with the grep -A flag I was having the below error grep: Not a recognized flag: A grep: Not a recognized flag: 1 Usage: grep [-r] [-R] [-H] [-L] [-E|-F] [-c|-l|-q] [-insvxbhwyu] [-p[parasep]] -e pattern_list... [-f pattern_file...] [file...] Usage: grep [-r] [-R] [-H] [-L] [-E|-F] [-c|-l|-q] [-insvxbhwyu] [-p[parasep]] [-e pattern_list...] -f pattern_file... [file...] Usage: grep [-r] [-R] [-H] [-L] [-E|-F] [-c|-l|-q] [-insvxbhwyu] [-p[parasep]] pattern_list [file...]
    – ramp
    Commented Oct 13, 2013 at 10:40
  • It is a GNU extension (I forgot to mention that) and for some reason I thought you were on Linux.
    – peterph
    Commented Oct 13, 2013 at 11:16
3

This command will assign the contents of one line to x, (agreed)

x=awk '/[0-9]_[0-9].*successful/{getline;getline;getline;print}' log filename | awk '{print $2}'
echo "$x"
2341

You process the entire file three times, but you could scan it once, and extract everything you need.

Suppose you process the entire file and count the lines. You used awk, but here is a script in perl, doing everything in a few lines,

#!/bin/env perl
use strict;
use warnings;

if( $#ARGV < 0 ) { die "no filename"; }
my ($batchname,$S_Time,$E_Time);
my ($success); #batch successul/failed?
my ($num,$pnum); #linenumber, processing line number
my ($fn,$fh,$line,@cols);

foreach $fn (@ARGV)
{
    if( !open($fh,"<$fn") ) { print "error: cannot open $fn\n"; next; }
    $success=0; $num=0; $pnum=-1; #line number
    while($line = <$fh>)
    {
        if( $line =~ /Batch.*\'([\d_]+)\'.*was successful/ )
        {
            $batchname = $1; $success = 1;
        }
        if( $line =~ /Processing batch \'([\d_]+)\'/ )
        {
            $batchname = $1; $pnum = $num;
        }
        if( $pnum > 0 ) {
        my @cols = split(/\s/,$line);
        if($num==$pnum) { $S_Time=$cols[1]; }
        if($num==$pnum+1) { $E_Time=$cols[1]; }
        }
        $num++;
    }
    if( $success ) { print "$batchname\t$S_Time\t$E_Time\n"; }
    else { print "$batchname encountered an error\n"; }
}

From the sample input/log file you provided:

2013/06/11 13:26:57 <0999> (725102)
Creating batch '12_2013162201.0'...
2013/06/11 13:26:57 <0999> (725102)
Batch '12_2013162201' was successful
2013/06/11 13:26:57 <0999> (725102)
TMR:Child ZERO, 160 Docs 320 Pgs 3874 KByts Tot 0.42 WAL 0.10 WALIO 0.15 IO 0.03 secs
2013/06/11 13:26:57 <0999> (725102)  Processing batch '12_2013162201'
2013/06/11 13:26:57 <0999> (725102)
Total in batch: 160 documents using 4 KBytes

Here is the output,

$ ./extractbatch.pl extractbatch.log 
12_2013162201   13:26:57        13:26:57

And here is the awk version (sorry about perl first, I'm a bit rusty on my awk),

#!/bin/awk -f
BEGIN {
    success=0; batchname="none";
    num=0; pnum=-1;
    S_Time="null"; E_Time="null";
}
#/Batch*was successful/ {
/was successful/ {
    batchname = $2; success = 1;
    printf "1: %s\n",batchname;
}
#/Processing batch */ {
/Processing/ {
    batchname = $7; pnum = num;
    printf "2: %s\n",batchname;
}
{
    if(num==pnum) S_Time=$2;
    if(num==pnum+1) E_Time=$2;
    num++;
}
END {
    if( success>0 ) { printf "%s\t%s\t%s\n",batchname,S_Time,E_Time; }
    else { printf "%s encountered an error\n",batchname; }
}

Note that the perl version does more.

2

This works, using the same basic ideas as your script:

#!/bin/ksh

fname=$1
for batchname in $(grep -i "Processing batch" $fname | cut -d "'" -f2)
do
    Batch_state=`grep -c -i "Batch '$batchname' was successful" $fname`
    if [[ "$Batch_state"  -ge 1 ]];then
    S_Time=$(grep "Processing.*$batchname" $fname | awk '{print $2}')
    E_Time=$(grep -A 1 "Processing.*$batchname" $fname | tail -n 1 | awk '{print $2}')
    echo -e "$batchname\t$S_Time\t$E_Time"
else
    echo $batchname encountered an error
fi
done
1
  • :It seems like -A flag is not recognized by ksh (I use ksh88). Because I am below error grep: Not a recognized flag: A grep: Not a recognized flag: 1 Usage: grep [-r] [-R] [-H] [-L] [-E|-F] [-c|-l|-q] [-insvxbhwyu] [-p[parasep]] -e pattern_list... [-f pattern_file...] [file...] Usage: grep [-r] [-R] [-H] [-L] [-E|-F] [-c|-l|-q] [-insvxbhwyu] [-p[parasep]] [-e pattern_list...] -f pattern_file... [file...] Usage: grep [-r] [-R] [-H] [-L] [-E|-F] [-c|-l|-q] [-insvxbhwyu] [-p[parasep]] pattern_list [file...]
    – ramp
    Commented Oct 13, 2013 at 10:36
0

You can use unset on your variables each time through your for loop. unset does what it says on the tin, un-sets a variable.

-bash-4.2$ export FOO=true
-bash-4.2$ echo $FOO
true
-bash-4.2$ unset FOO
-bash-4.2$ echo $FOO

-bash-4.2$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .