Reusing a variable

Question

I was breaking my head for a day still not successful.

I have this script:

#!/bin/ksh

fname=$1

for batchname in $(grep -i "Processing batch" $fname | cut -d "'" -f2)
do
Batch_state=`grep -c -i "Batch '$batchname' was successful" $fname`
if [[ "$Batch_state"  -ge 1 ]];then
{
S_Time=`awk '/[0-9]_[0-9].*successful/{getline;getline;getline;print}' $fname | awk '{print $2}'`
E_Time=`awk '/[0-9]_[0-9].*successful/{getline;getline;getline;getline;print}' $fname | awk '{print $2}'`
echo -e $batchname"\t"$S_Time"\t"$E_Time
}
else
{
echo $batchname encountered an error
}
fi
done

Output this code is producing:

02_1231324      14:29:04 15:29:11   14:32:19 15:33:11
79_3097935      14:29:04 15:29:11   14:32:19 15:33:11

Desired Output:

02_1231324      14:29:04    14:32:19
79_3097935      15:29:11    15:33:11 

Sample Input:

2013/06/11 14:29:04 <0999> (725102)
Creating batch '02_1231324.0'...
2013/06/11 14:29:04 <0999> (725102)
Batch '02_1231324' was successful
2013/06/11 14:29:04 <0999> (725102)
TMR:Child ZERO, 160 Docs 320 Pgs 3874 KByts Tot 0.42 WAL 0.10 WALIO 0.15 IO 0.03 secs
2013/06/11 14:29:04 <0999> (725102)  Processing batch '02_1231324'
2013/06/11 14:32:19 <0999> (725102)
Total in batch: 160 documents using 4 KBytes
2013/06/11 15:29:11 <0999> (725102)
Creating batch '79_3097935.0'...
2013/06/11 15:29:11 <0999> (725102)
Batch '79_3097935' was successful
2013/06/11 15:29:11 <0999> (725102)
TMR:Child ZERO, 160 Docs 320 Pgs 3874 KByts Tot 0.42 WAL 0.10 WALIO 0.15 IO 0.03 secs
2013/06/11 15:29:11 <0999> (725102)  Processing batch '79_3097935'
2013/06/11 15:33:11 <0999> (725102)
Total in batch: 160 documents using 4 KBytes
TMR:Child ZERO, 160 Docs 320 Pgs 3874 KByts Tot 0.42 WAL 0.10 WALIO 0.15 IO 0.03 secs
2013/06/11 13:26:57 <0999> (725102)  Processing batch '12_2013162201'
2013/06/11 13:26:57 <0999> (725102)
Total in batch: 160 documents using 4 KBytes

So, what is wrong with my script? How can I get the desired output?

Answer 1

You probably want something like this to parse your log:

#!/bin/awk -f
{
    if (/was successful/) {
        bn = $2;
        gsub(/'/, "", bn);
        succ[bn] = 1;
    }
    if (/Processing batch/) {
        bn = $7;
        gsub(/'/, "", bn);
        if (bn in succ) {
            succ[bn,",s"] = $2;
            getline
            succ[bn,",e"] = $2;
        } else
            fail[bn] = 1;
    }
}
END {
    for (val in succ)
        if (val !~ /,/)
            print val " " succ[val,",s"] " " succ[val,",e"];
    for (val in fail)
        print val " encountered an error";
}

---

Your original script slightly reduced to just 1 awk call under some strict¹ assumptions about the structure of your input data).

#!/bin/ksh

fname=$1

for batchname in $(grep -i "Processing batch" $fname | cut -d "'" -f2); do
    Batch_state=`grep -c -i "Batch '$batchname' was successful" $fname`
    if [[ "$Batch_state"  -ge 1 ]]; then
        awk -v b=$batchname '
{
    if ($0 ~ "Processing.*" b) {
        s = $2;
        getline;
        e = $2;
        print b " " s " " e;
        exit
    }
}' $fname
    else
        echo $batchname encountered an error
    fi
done

Shorter version using grep -A (-A is a GNU extension which selects the matching line and a number - 1 by default - of lines that follow):

#!/bin/ksh

fname=$1

for batchname in $(grep -i "Processing batch" $fname | cut -d "'" -f2); do
    Batch_state=`grep -c -i "Batch '$batchname' was successful" $fname`
    printf "%s" $batchname
    if [[ "$Batch_state"  -ge 1 ]]; then
        grep -A1 "Processing batch '$batchname'" $fname | cut -f2 -d" " | fmt
    else
        printf "encountered an error\n"
    fi
done

But these are still processing the file 3 times. See ChuckCottrill's answer for how it could be done.

---

¹ it seems that the batches are not processed in parallel, and what you want is actually the line containing Processing batch and the line after that.

Answer 2

This command will assign the contents of one line to x, (agreed)

x=awk '/[0-9]_[0-9].*successful/{getline;getline;getline;print}' log filename | awk '{print $2}'
echo "$x"
2341

You process the entire file three times, but you could scan it once, and extract everything you need.

Suppose you process the entire file and count the lines. You used awk, but here is a script in perl, doing everything in a few lines,

#!/bin/env perl
use strict;
use warnings;

if( $#ARGV < 0 ) { die "no filename"; }
my ($batchname,$S_Time,$E_Time);
my ($success); #batch successul/failed?
my ($num,$pnum); #linenumber, processing line number
my ($fn,$fh,$line,@cols);

foreach $fn (@ARGV)
{
    if( !open($fh,"<$fn") ) { print "error: cannot open $fn\n"; next; }
    $success=0; $num=0; $pnum=-1; #line number
    while($line = <$fh>)
    {
        if( $line =~ /Batch.*\'([\d_]+)\'.*was successful/ )
        {
            $batchname = $1; $success = 1;
        }
        if( $line =~ /Processing batch \'([\d_]+)\'/ )
        {
            $batchname = $1; $pnum = $num;
        }
        if( $pnum > 0 ) {
        my @cols = split(/\s/,$line);
        if($num==$pnum) { $S_Time=$cols[1]; }
        if($num==$pnum+1) { $E_Time=$cols[1]; }
        }
        $num++;
    }
    if( $success ) { print "$batchname\t$S_Time\t$E_Time\n"; }
    else { print "$batchname encountered an error\n"; }
}

From the sample input/log file you provided:

2013/06/11 13:26:57 <0999> (725102)
Creating batch '12_2013162201.0'...
2013/06/11 13:26:57 <0999> (725102)
Batch '12_2013162201' was successful
2013/06/11 13:26:57 <0999> (725102)
TMR:Child ZERO, 160 Docs 320 Pgs 3874 KByts Tot 0.42 WAL 0.10 WALIO 0.15 IO 0.03 secs
2013/06/11 13:26:57 <0999> (725102)  Processing batch '12_2013162201'
2013/06/11 13:26:57 <0999> (725102)
Total in batch: 160 documents using 4 KBytes

Here is the output,

$ ./extractbatch.pl extractbatch.log 
12_2013162201   13:26:57        13:26:57

And here is the awk version (sorry about perl first, I'm a bit rusty on my awk),

#!/bin/awk -f
BEGIN {
    success=0; batchname="none";
    num=0; pnum=-1;
    S_Time="null"; E_Time="null";
}
#/Batch*was successful/ {
/was successful/ {
    batchname = $2; success = 1;
    printf "1: %s\n",batchname;
}
#/Processing batch */ {
/Processing/ {
    batchname = $7; pnum = num;
    printf "2: %s\n",batchname;
}
{
    if(num==pnum) S_Time=$2;
    if(num==pnum+1) E_Time=$2;
    num++;
}
END {
    if( success>0 ) { printf "%s\t%s\t%s\n",batchname,S_Time,E_Time; }
    else { printf "%s encountered an error\n",batchname; }
}

Note that the perl version does more.

Answer 3

This works, using the same basic ideas as your script:

#!/bin/ksh

fname=$1
for batchname in $(grep -i "Processing batch" $fname | cut -d "'" -f2)
do
    Batch_state=`grep -c -i "Batch '$batchname' was successful" $fname`
    if [[ "$Batch_state"  -ge 1 ]];then
    S_Time=$(grep "Processing.*$batchname" $fname | awk '{print $2}')
    E_Time=$(grep -A 1 "Processing.*$batchname" $fname | tail -n 1 | awk '{print $2}')
    echo -e "$batchname\t$S_Time\t$E_Time"
else
    echo $batchname encountered an error
fi
done

Answer 4

You can use unset on your variables each time through your for loop. unset does what it says on the tin, un-sets a variable.

-bash-4.2$ export FOO=true
-bash-4.2$ echo $FOO
true
-bash-4.2$ unset FOO
-bash-4.2$ echo $FOO

-bash-4.2$