0

I have a log file whose name is listlog, the content is:

2021-08-12 16:09:17 textsp sdgg
reponse:success
prams:invalid
line 3
2021-08-12 16:10:17 textdfdfdlfs sfdfs
reponse: failed
prams:valid
line 3

I want to filter the logs recorded between t1 and t2 and print the log line with 2 consecutive lines. I wrote using awk with if condition and grep. But I got the error:

This is the content I want to filter and write to a new file assuming t1 < (16:09:17, 16:19:17) < t2 :

2021-08-12 16:09:17 textsp sdgg
reponse:success
prams:invalid
2021-08-12 16:10:17 textdfdfdlfs sfdfs
reponse: failed
prams:valid

here is the awk script i wrote:

t1=$(date -d "$min_change minutes ago" +"%F %T")
t2=$(date +"%F %T")
cat $listlog | awk -v t1="$t1" -v t2="$t2" -v listlog="$listlog" '{
        if (match($0 ~ /^[0-9]{4}(-[0-9]{2}){2}))
        {
                if ( $0 > t1 && $0 < t2 | $0 ~ t2 )
                {
                        grep -A 2 $0 $listlog >> /tmp/temp.txt
                }
        }
        }'

This is the error I get:

awk: cmd. line:2:       if (match($0 ~ /^[0-9]{4}(-[0-9]{2}){2}))
awk: cmd. line:2:                       ^ unterminated regexp
awk: cmd. line:3:       if (match($0 ~ /^[0-9]{4}(-[0-9]{2}){2}))
awk: cmd. line:3:                                                ^ unexpected newline or end of string
awk: cmd. line:4:               if ( $0 > t1 && $0 < t2 | $0 ~ t2 )
awk: cmd. line:4:                                         ^ syntax error.
awk: cmd. line:5:               if ( $0 > t1 && $0 < t2 | $0 ~ t2 ) 
awk: cmd. line:5:                                                   ^ unexpected newline or end of string
awk: cmd. line:6:                       grep -A 2 $0 $listlog >> /temp/temp.txt
awk: cmd. line:6:                                             ^ syntax error
awk: cmd. line:7:                       grep -A 2 $0 $listlog >> /temp/temp.txt.
awk: cmd. line:7:                                                                ^ unexpected newline or end of string
2

1 Answer 1

2

It's really hard to even begin describing what's wrong with your script, but here's a summary of the most important points:

  1. What you've written isn't an awk script. it's not even an awk script embedded within a shell script (which would normally be perfectly OK). It's a weird mixture of some shell code, with an attempt at an awk script that includes shell code within it (awk doesn't interpret or run shell commands).

    You can't run grep like that inside awk, not without forking it as an external process, anyway. And there's no need to do that because awk can do regex pattern matching.

  2. You have no indentation in any of your code, making it extremely difficult to read.

    UPDATE: I just edited your question to re-format it, and it turns out you did have some indentation in your code. But you'd formatted it with HTML <br/> tags instead of using the markdown editor's built-in text formatting abilities.

  3. Points 1 and 2, are probably why you've had no answers or even comments in the 7 hours since you posted your question. The initial reaction of any reader (mine, at least) would be "WTF? That makes no sense. I don't even know where to begin...pass".

  4. You can't do date comparisons in shell or awk the way you're attempting to do them. To do numeric comparisons like greater-than or less-than, you need to convert the dates to numbers (i.e. unix time_t format, seconds since the "epoch", which is Midnight on Jan 1 1970, UTC).

  5. It is possible to do what you want in awk, but it'll be harder than necessary because awk doesn't have any built-in date conversion functions. It would be significantly easier in perl, which does have good date & time library functions. Which is a good thing for you, because your script as written is much closer to perl code in style than awk code.


Here's a fairly simple, straight-forward way of doing what you want in perl. It takes command line arguments (using perl's Getopt::Std library module). Getopt::Std is a core perl library module and is included with perl.

The arguments can be provided in any order. None of the option arguments (-m, -n, and -s) are mandatory, they all have default values if not provided on the command line. Any args which aren't handled by Getopt::Std are treated as input filenames. The script can process stdin as well as any input filenames, so you can pipe your log file into it.

The script also uses the Date::Parse module from the perl Time::Date collection to convert date strings to seconds since the epoch.

Date::Parse is not included with perl, it needs to be installed. If you're using Debian (or Ubuntu or Mint or other Debian-related distros) you run install it with sudo apt-get install libtimedate-perl. Most other distros will have it packaged too. The Centos package is perl-TimeDate-2.30-2 for Centos 7, or perl-TimeDate-2.30-15 for Centos 8. Otherwise, you can install it with cpan.

#!/usr/bin/perl
use strict;
use Getopt::Std;
use Date::Parse;

my %opts;
getopt('m:s:n:', \%opts); # -m minutes, -s start time, -n number of lines

my $st  = $opts{s} || -1; # start time, default now.
my $min = $opts{m} ||  5; # number of minutes after $st to match, default 5.
my $nl  = $opts{n} ||  2; # No. of extra lines to print after match, default 2.

if ($st == -1) {
  $st = time();           # now
} else {
  $st = str2time($st);    # convert time string to seconds since epoch
};
my $et = $st + $min * 60; # calculate ending time

my $p = -1; # input lines are printed only when $p > -1

# main loop, read and process all input file(s).
while (<>) {
  if (m/^(\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d)/) {
    my $t = str2time($1);
    $p = $nl if ($t >= $st && $t <= $et);
  };

  if ($p > -1) { print; $p-- };
};

Save it as, e.g. extract.pl, make it executable with chmod +x extract.pl and run it like this:

$ ./extract.pl -s "2021-08-12 16:09:00" -n 2 -m 5 listlog 
2021-08-12 16:09:17 textsp sdgg
reponse:success
prams:invalid
2021-08-12 16:10:17 textdfdfdlfs sfdfs
reponse: failed
prams:valid

You can redirect the output if required, e.g. by appending >>/tmp/temp.txt to the end of the command line.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .