Logs for the last n minutes

Question

I am looking for getting the logs for the last n minutes. I tried a few options but most of them didn't work if there are no logs specified for a particular timestamp.

For example:
The one worked fine if current time is 2019-07-27 09:00:01, and I look for the log 10 minutes ago till current time. It worked if there is a line in the log file 2019-07-27 08:50:01, but it doesn't print anything if there is no line in logs at this timestamp.

for (( i = 10; i >=0; i-- )) ; do grep $(date +%R -d "-$i  min") test.log

I also tried an awk option given in the thread Get logs for last 30 minutes

However it didn't work whenever there is a line in the log file that does not have a date timestamp in the beginning.

awk -vStartDate=`date -d'now-30 min' +%H:%M:%S` '{ if ($2 > StartDate) print $0}' test.log

Sample lines in my logfile are:

2019-07-27 01:28:35,8291 DEBUG [TestLogger] - Parameters: [100, IN%]
   2019-07-27 01:28:35,8292 ERROR [TestLogger] - Query with error:
SELECT COUNT(*) FROM <table_name>

   2019-07-27 01:28:35,8293 ERROR [TestLogger] - 'Connection reset' while executing statement
   2019-07-27 01:28:35,8294 WARN [TestReportAction] - Count Error: TestReportAction - 'Connection reset' while executing statement

It should trim the line number as well while matching the time which is given after a comma ,.

Answer 1

If you have both perl and the Date::Parse perl module installed, you can do it with:

perl -MDate::Parse -F, -a -n -e \
  'BEGIN { $start=time() - 600 ; $p=0};
   if ($p) { print } else {
     $logtime = str2time($F[0]);
     $p=1 if ($logtime >= $start)}' /path/to/logfile

This perl one-liner assumes the field-delimiter between the logfile's timestamp and the remainder of the data is a comma (-F,) and uses perl's auto-split option (-a) to autosplit each input line into the @F array.

It uses the variable $p as a toggle switch for printing to be on or off, defaulting to off (0). If $p is on then print the current line, otherwise use Date::Parse's str2time() function to convert the log entry's timestamp to a time_t value (seconds sine the epoch, 1970-01-01 00:00:00) and then check whether the log line's timestamp is >= the desired start time (600 seconds, 10 minutes, ago). If it is, set $p to on (1).

If you have more complex requirements, it's probably better to write a standalone perl script rather than try to do it as a one-liner - editing on the bash command line is a PITA compared to using a text editor. You could start by converting the above one-liner:

#!/usr/bin/perl

use Date::Parse;

$start = time() - 600;
$p = 0;

while (<>) {
  if ($p) {
    print
  } else {
    my (@F) = split /,/;
    my $logtime = str2time($F[0]);
    $p = 1 if ($logtime >= $start);
  }
}

e.g. it would be fairly easy to modify this script to take the starting time from the first command line argument, instead of hard-coding it to -600 seconds.

This could be done in awk instead of perl, but perl's Date::Parse module is good and easy to use, a lot easier than writing date parsing code in awk. Date::Parse is easily installed from CPAN and is available packaged for some linux distros (e.g. in the libtimedate-perl package on debian).

The same basic algorithm could also be written in python or any other language with a decent date parsing function.