Extract X number of minutes from the current time from a logfile

Question

I have a logfile with the following timestamp format

2016-06-02 13:59:58.069

The log I am working with spans about an hour.

2016-06-02 13:05:06.144
2016-06-02 13:05:06.144
2016-06-02 13:05:06.160
2016-06-02 13:05:06.160
2016-06-02 13:05:06.176
2016-06-02 13:05:06.177
.
.
.
.
2016-06-02 14:05:03.033
2016-06-02 14:05:03.034
2016-06-02 14:05:03.034
2016-06-02 14:05:03.084
2016-06-02 14:05:03.096
2016-06-02 14:05:03.112

When I try to use awk to extract out the last 10 minutes I still get the whole file.

awk -vDate=`date -d'now-10 minutes' +%Y/%m/%d:%H:%M:%S` '$1,$2 >   Date {print Date, $0}' logfile.log | less

Awk output:

2016/06/02:08:57:35 2016-06-02 13:05:06.144
. 
.
.
2016/06/02:08:57:35 2016-06-02 14:08:05.214

What am I doing wrong?

Answer 1

Using a different approach. I get the vDate, grep to obtain the line number and tail the file from that line number :

vDate=`date -d'now-10 minutes' '+%Y-%m-%d %H:%M'` 
line=`grep  -n "$vDate" logfile.log | head -1  | cut -d: -f1`
tail -n +${line} logfile.log

Answer 2

Try

awk -vDate="`date -d'TZ="UTC" now-10 minutes' +'%Y-%m-%d %H:%M:%S.%3N'`" '$1" "$2 > Date {print Date, $0}'

In the pattern $1,$2 > Date, the comma doesn't mean "join these fields" like it does in the output; instead, it means the pattern is a record range. $1 is a non-empty string, which is always true, so the match starts immediately. You need to put a literal space there to join the fields before comparing: $1" "$2

Next, you can see in your example output that you were setting Date to "2016/06/02:...", while the logs used "2016-06-02 ...". As you are comparing date strings, and / is later than - in ASCII, the test would never succeed. Changing the string format to match exactly (with some added quoting to protect it from the shell) means only the digits will differ.

Edit: Your logfile timestamps are a few hours later than what date -d'now-10 minutes' shows. You can tell date to use the relevant timezone either with an environment variable (TZ=UTC awk ...) or inside the date string 'TZ="UTC" now...'. If it's not UTC, pick the right one from the complete list.

Answer 3

awk can print the lines of a text in a section with start_pattern and end_pattern, use the following syntax:

$ awk '/start_pattern/, /end _pattern/' filename

quote from Linux Shell Scripting Cookbook.

So, you can do it like this:

awk '/2016-06-02 13:55/, /2016-06-02 14:05/' logfile`

You could also writing this to the script file if you want:

START=`date --date='now-10 minutes' '+%Y-%m-%d %H:%M'`
END=`date --date='now' '+%Y-%m-%d %H:%M'`
awk "/$START/, /$END/" logfile