0

I am looking for getting the logs for the last n minutes. I tried a few options but most of them didn't work if there are no logs specified for a particular timestamp.

For example:
The one worked fine if current time is 2019-07-27 09:00:01, and I look for the log 10 minutes ago till current time. It worked if there is a line in the log file 2019-07-27 08:50:01, but it doesn't print anything if there is no line in logs at this timestamp.

for (( i = 10; i >=0; i-- )) ; do grep $(date +%R -d "-$i  min") test.log

I also tried an awk option given in the thread Get logs for last 30 minutes

However it didn't work whenever there is a line in the log file that does not have a date timestamp in the beginning.

awk -vStartDate=`date -d'now-30 min' +%H:%M:%S` '{ if ($2 > StartDate) print $0}' test.log

Sample lines in my logfile are:

2019-07-27 01:28:35,8291 DEBUG [TestLogger] - Parameters: [100, IN%]
   2019-07-27 01:28:35,8292 ERROR [TestLogger] - Query with error:
SELECT COUNT(*) FROM <table_name>

   2019-07-27 01:28:35,8293 ERROR [TestLogger] - 'Connection reset' while executing statement
   2019-07-27 01:28:35,8294 WARN [TestReportAction] - Count Error: TestReportAction - 'Connection reset' while executing statement

It should trim the line number as well while matching the time which is given after a comma ,.

  • If you are able to use journalctl you can do something like journalctl --since '10 minutes ago' – Valentin Bajrami Jul 27 at 18:37
  • Unfortunately, jounalctl is not an option. – Nik Jul 28 at 1:22
0

If you have both perl and the Date::Parse perl module installed, you can do it with:

perl -MDate::Parse -F, -a -n -e \
  'BEGIN { $start=time() - 600 ; $p=0};
   if ($p) { print } else {
     $logtime = str2time($F[0]);
     $p=1 if ($logtime >= $start)}' /path/to/logfile

This perl one-liner assumes the field-delimiter between the logfile's timestamp and the remainder of the data is a comma (-F,) and uses perl's auto-split option (-a) to autosplit each input line into the @F array.

It uses the variable $p as a toggle switch for printing to be on or off, defaulting to off (0). If $p is on then print the current line, otherwise use Date::Parse's str2time() function to convert the log entry's timestamp to a time_t value (seconds sine the epoch, 1970-01-01 00:00:00) and then check whether the log line's timestamp is >= the desired start time (600 seconds, 10 minutes, ago). If it is, set $p to on (1).

If you have more complex requirements, it's probably better to write a standalone perl script rather than try to do it as a one-liner - editing on the bash command line is a PITA compared to using a text editor. You could start by converting the above one-liner:

#!/usr/bin/perl

use Date::Parse;

$start = time() - 600;
$p = 0;

while (<>) {
  if ($p) {
    print
  } else {
    my (@F) = split /,/;
    my $logtime = str2time($F[0]);
    $p = 1 if ($logtime >= $start);
  }
}

e.g. it would be fairly easy to modify this script to take the starting time from the first command line argument, instead of hard-coding it to -600 seconds.

This could be done in awk instead of perl, but perl's Date::Parse module is good and easy to use, a lot easier than writing date parsing code in awk. Date::Parse is easily installed from CPAN and is available packaged for some linux distros (e.g. in the libtimedate-perl package on debian).

The same basic algorithm could also be written in python or any other language with a decent date parsing function.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.