Select everything between two timestamps in a log file

Question

awk '/10:..:/, /13:..:/' server.log > /tmp/awktmp

I tried this command as someone gave me. It worked for me to find logs between 10:00 AM to 13:00 PM but I don't understand it fully.

Please tell me an elegant solution if you've one. Remember the remote servers are minimal and don't have advanced utilities.

What I don't understand here is '/10:..:/, /13:..:/'

• I understand 10:.. means 10 and : and match exactly 2 digits.
• But what does another : mean?

I suppose the forwardslashes are for putting regular expression like we do in JavaScript regex. I'm not 100% sure about that.

Update 1:

Time is in second column, in format HH:MM:SS

Update 2:

sed -n -e '/8:..:../,/9:..:../p' application.log > /tmp/sedtmp

I tried this but this is also showing logs from 07:57:47.

Update 3

The logs don't always contain the timestamp in each row. Some rows don't contain any timestamp at all. How'd you overcome that issue? I stupidly used awk and I missed all the rows which didn't have time value in them. Is there a way to avoid this issue?

Like this is what I mean.

2023-08-07 09:20:35 0123456789 INFO  CustomerLogoutResource:95 - Entering logout api.
2023-08-07 09:20:35 0123456789 ERROR AppExceptionMapper:87 - Exception has been thrown by container
2023-08-07 09:20:35 0123456789 ERROR AppExceptionMapper:555 - Unchecked Exception
java.lang.NullPointerException
    at NullPointerExceptionExample.printLength(NullPointerExceptionExample.java:3)
    at NullPointerExceptionExample.main(NullPointerExceptionExample.java:8)

Answer 1

That kind of approach fails to find the lines for 11:00 to 13:00 if there are no log inbetween 10:00 and 11:00 and would report all lines after 14:00 if there was no log in between 13:00 and 14:00 (and there was at least one between 10 and 11).

Best for this kind of thing it to do lexical comparison of the the time against the boundaries.

For instance, if the time is in the third field:

awk '$3 >= "10:00:00" && $3 < "13:00:00"'

If you don't know where in the line the time is, you could do:

perl -lne 'print if /\b\d\d:\d\d:\d\d\b/ &&
                      $& ge "10:00:00" &&
                      $& lt "13:00:00"'

Or:

LC_ALL=C awk 'match($0, /[0-2][0-9]:[0-5][0-9]:[0-5][0-9]/) && \
                (t = substr($0, RSTART, 8)) >= "10:00:00" && \
                t < "13:00:00"'

Those report the lines that have a timestamp in the range. If you have lines without timestamps in between lines with timestamps and you want them reported, you can use the beginning-condition, end-condition {action} approach like in your question but again use comparison instead of regex match, or do the state switching by hand so as to keep excluding the upper boundary:

LC_ALL=C awk -v beg=10:00:00 -v end=13:00:00 '
  match($0, /[0-2][0-9]:[0-5][0-9]:[0-5][0-9]/) {
    t = substr($0, RSTART, 8)
    if (t >= end) selected = 0
    else if (t >= beg) selected = 1
  }
  selected'

Answer 2

The pattern /10:..:/ matches the digits 10, two separators of :, and any two characters between them. So it will match a time (anywhere within the input line) like 10:35:22. But it would also match a line containing This10:ZZ:Camels, so it is not a very good test.

A better pattern might be /10:[0-5][0-9]:[0-5][0-9]/, which checks that the minutes and seconds are in the range 00-59. But it could also be helpful to check the times are in a particular field, or have whitespace around then, or are (inclusively)near the start of the record. You might post a few sample input lines so we can better tell what is required.

Having two patterns separated by a comma "switches on" the match when the first pattern is detected, and "switches off" the match when the second pattern is detected. It will match all lines between those events (inclusively), even if they do not contain a date at all.

This is very different to the single pattern /1[0-2]:[0-5][0-9]:[0-5][0-9]/, which would match only individual lines between 10:00:00 and 12:59:59, in whatever order they happened to be.

Answer 3

the awk pattern:

/regA/,/regB/

will be true from the first line matching regA, until the first line matching regB. And as usual, when something matches an expression: if there are no { actions } after the expression to precise what actions to perform when it matches, the default action is : print the line where the expression is true.

Now they chose:

/10:..:/, /13:..:/

to be a (little) bit more sure that you match 10:mm:ss and not hh:10:ss. (They do this by assuming that any citation of time will be : hh:mm:ss, and that there are no ':' before the hour ... this is not always true, depending on the date format used. And as said in the comments, it could also match other things as well.

If you know that the beginning of lines is always:

YYYY-MM-DDThh:mm:ss+hh:mm 
# For exemple:
# 2023-07-17T11:14:02+02:00 , which is following internationnal recommendations
# of displaying date and time (and timezone)

You could match more closely with:

/^2023-07-17T10:/,/^2023-07-17T13:/

and ensure you match only from the beginning (^) of the line and not wherever those things could appear on the line.

Answer 4

Here is a possible solution that selects logs between two time stamps in a log file. This script assumes that the log file has space-separated columns, and the second column contains the time stamp in the format HH:MM:SS. Might not be the most elegant but at least it is legible.

awk -F" " '$2 >= "09:00:00" && $2 <= "12:00:00"' server.log

• -F" " sets the field separator to a space character.
• $2 refers to the second field in each line of the input file (fields are separated by spaces in this case).
• >= "09:00:00" checks if the value of the second field is greater than or equal to "09:00:00".
• && a logical operator that combines conditions. It means "and"
• $2 <= "12:00:00" checks if the value of the second field is less than or equal to "12:00:00".
• sever.log is your input file

Solution Verification

For a server.log that looks something like this:

2023-07-18 08:55:32 - Log entry 1
2023-07-18 09:10:15 - Log entry 2
2023-07-18 10:30:47 - Log entry 3
2023-07-18 11:45:02 - Log entry 4
2023-07-18 12:05:21 - Log entry 5
2023-07-18 13:20:33 - Log entry 6

Output:

[get@me test]$ awk -F" " '$2 >= "09:00:00" && $2 <= "12:00:00"' server.log
2023-07-18 09:10:15 - Log entry 2
2023-07-18 10:30:47 - Log entry 3
2023-07-18 11:45:02 - Log entry 4