5
awk '/10:..:/, /13:..:/' server.log > /tmp/awktmp

I tried this command as someone gave me. It worked for me to find logs between 10:00 AM to 13:00 PM but I don't understand it fully.

Please tell me an elegant solution if you've one. Remember the remote servers are minimal and don't have advanced utilities.

What I don't understand here is '/10:..:/, /13:..:/'

  • I understand 10:.. means 10 and : and match exactly 2 digits.
  • But what does another : mean?

I suppose the forwardslashes are for putting regular expression like we do in JavaScript regex. I'm not 100% sure about that.

Update 1:

Time is in second column, in format HH:MM:SS

Update 2:

sed -n -e '/8:..:../,/9:..:../p' application.log > /tmp/sedtmp

I tried this but this is also showing logs from 07:57:47.

Update 3

The logs don't always contain the timestamp in each row. Some rows don't contain any timestamp at all. How'd you overcome that issue? I stupidly used awk and I missed all the rows which didn't have time value in them. Is there a way to avoid this issue?

Like this is what I mean.

2023-08-07 09:20:35 0123456789 INFO  CustomerLogoutResource:95 - Entering logout api.
2023-08-07 09:20:35 0123456789 ERROR AppExceptionMapper:87 - Exception has been thrown by container
2023-08-07 09:20:35 0123456789 ERROR AppExceptionMapper:555 - Unchecked Exception
java.lang.NullPointerException
    at NullPointerExceptionExample.printLength(NullPointerExceptionExample.java:3)
    at NullPointerExceptionExample.main(NullPointerExceptionExample.java:8)
1
  • Can you check the logs carefully? Three possible causes. (a) The lines with 07:57:47 may have something else in the same line that does match (remember awk knows about columns, but sed examines the whole line). (b) The logged messages may not be in precise time order (they may be buffered, or come from systems with clock drift), so there may be a previous line that did have 08:??:??. (c) A line from the previous day for 18:??:?? may have triggered the range, because you still omit the leading zero on the hour. Cannot you just cut columns 1 and 2 and post those to PasteBin ? Jul 18, 2023 at 12:19

4 Answers 4

8

That kind of approach fails to find the lines for 11:00 to 13:00 if there are no log inbetween 10:00 and 11:00 and would report all lines after 14:00 if there was no log in between 13:00 and 14:00 (and there was at least one between 10 and 11).

Best for this kind of thing it to do lexical comparison of the the time against the boundaries.

For instance, if the time is in the third field:

awk '$3 >= "10:00:00" && $3 < "13:00:00"'

If you don't know where in the line the time is, you could do:

perl -lne 'print if /\b\d\d:\d\d:\d\d\b/ &&
                      $& ge "10:00:00" &&
                      $& lt "13:00:00"'

Or:

LC_ALL=C awk 'match($0, /[0-2][0-9]:[0-5][0-9]:[0-5][0-9]/) && \
                (t = substr($0, RSTART, 8)) >= "10:00:00" && \
                t < "13:00:00"'

Those report the lines that have a timestamp in the range. If you have lines without timestamps in between lines with timestamps and you want them reported, you can use the beginning-condition, end-condition {action} approach like in your question but again use comparison instead of regex match, or do the state switching by hand so as to keep excluding the upper boundary:

LC_ALL=C awk -v beg=10:00:00 -v end=13:00:00 '
  match($0, /[0-2][0-9]:[0-5][0-9]:[0-5][0-9]/) {
    t = substr($0, RSTART, 8)
    if (t >= end) selected = 0
    else if (t >= beg) selected = 1
  }
  selected'
10
  • 1
    With a POSIX awk for RE intervals you could make [0-2][0-9]:[0-5][0-9]:[0-5][0-9] slightly briefer as [0-2][0-9](:[0-5][0-9]){2} but obviously no big deal either way. FWIW I find tests for a value inside a range slightly clearer written as '"10:00:00" <= $3 && $3 < "13:00:00"' rather than '$3 >= "10:00:00" && $3 < "13:00:00"' so the code mimics and highlights the functionality that $3 is between the other values and so you don't even have to think about what it means when you see it but again nbd.
    – Ed Morton
    Jul 17, 2023 at 14:39
  • @Stephen. It didn't work... awk '$2 >="8:00:00" && $2<="9:00:00"' application.log >awkError
    – achhainsan
    Jul 18, 2023 at 3:36
  • @achhainsan We still have not seen any actual input data, and I can't guess what "didn't work" means -- did a Bird of Paradise fly up your nose ? But obviously 8:00:00 will match both 08:00:00 and 18:00:00, so maybe that's it. Jul 18, 2023 at 7:29
  • I've updated how the logs look like. Take a look.
    – achhainsan
    Aug 7, 2023 at 3:36
  • 1
    Then the last approach should work for your input. Aug 7, 2023 at 5:02
8

The pattern /10:..:/ matches the digits 10, two separators of :, and any two characters between them. So it will match a time (anywhere within the input line) like 10:35:22. But it would also match a line containing This10:ZZ:Camels, so it is not a very good test.

A better pattern might be /10:[0-5][0-9]:[0-5][0-9]/, which checks that the minutes and seconds are in the range 00-59. But it could also be helpful to check the times are in a particular field, or have whitespace around then, or are (inclusively)near the start of the record. You might post a few sample input lines so we can better tell what is required.

Having two patterns separated by a comma "switches on" the match when the first pattern is detected, and "switches off" the match when the second pattern is detected. It will match all lines between those events (inclusively), even if they do not contain a date at all.

This is very different to the single pattern /1[0-2]:[0-5][0-9]:[0-5][0-9]/, which would match only individual lines between 10:00:00 and 12:59:59, in whatever order they happened to be.

5
  • what's with [0-5] [0-9] in MM column and SS column?
    – achhainsan
    Jul 18, 2023 at 5:02
  • @achhainsan It matches two consecutive characters which make up the range 00 to 59, just like my answer says. So it checks that the MM and SS parts are actually numeric, and in the correct range for those fields. It is wise to minimise false matches (like 10:ZZ: in my answer), especially as we have not seen any of your actual data. In particular, log records tend to be effusive and contain strings that come from user data input which can contain almost anything. Jul 18, 2023 at 7:38
  • 1
    But it would also match a line containing This10:ZZ:Camels, so it is not a very good test. Which is just another reason to always make the first field of any log file an ISO8601-formatted date/time. Jul 18, 2023 at 15:52
  • @AndrewHenle Completely agree. But a lot of legacy applications just use their date local default, and then start scraping logs when the systems get buggy. I even found one that left the date off completely -- we counted backwards to find time transitions over 00:00:00, and devoutly hoped there were no hidden days. Jul 18, 2023 at 16:56
  • I've updated how the logs look like. Take a look.
    – achhainsan
    Aug 7, 2023 at 3:37
1

the awk pattern:

/regA/,/regB/

will be true from the first line matching regA, until the first line matching regB. And as usual, when something matches an expression: if there are no { actions } after the expression to precise what actions to perform when it matches, the default action is : print the line where the expression is true.

Now they chose:

/10:..:/, /13:..:/

to be a (little) bit more sure that you match 10:mm:ss and not hh:10:ss. (They do this by assuming that any citation of time will be : hh:mm:ss, and that there are no ':' before the hour ... this is not always true, depending on the date format used. And as said in the comments, it could also match other things as well.

If you know that the beginning of lines is always:

YYYY-MM-DDThh:mm:ss+hh:mm 
# For exemple:
# 2023-07-17T11:14:02+02:00 , which is following internationnal recommendations
# of displaying date and time (and timezone)

You could match more closely with:

/^2023-07-17T10:/,/^2023-07-17T13:/

and ensure you match only from the beginning (^) of the line and not wherever those things could appear on the line.

1
  • I've updated how the logs look like. Take a look.
    – achhainsan
    Aug 7, 2023 at 3:36
1

Here is a possible solution that selects logs between two time stamps in a log file. This script assumes that the log file has space-separated columns, and the second column contains the time stamp in the format HH:MM:SS. Might not be the most elegant but at least it is legible.

awk -F" " '$2 >= "09:00:00" && $2 <= "12:00:00"' server.log
  • -F" " sets the field separator to a space character.
  • $2 refers to the second field in each line of the input file (fields are separated by spaces in this case).
  • >= "09:00:00" checks if the value of the second field is greater than or equal to "09:00:00".
  • && a logical operator that combines conditions. It means "and"
  • $2 <= "12:00:00" checks if the value of the second field is less than or equal to "12:00:00".
  • sever.log is your input file

Solution Verification

For a server.log that looks something like this:

2023-07-18 08:55:32 - Log entry 1
2023-07-18 09:10:15 - Log entry 2
2023-07-18 10:30:47 - Log entry 3
2023-07-18 11:45:02 - Log entry 4
2023-07-18 12:05:21 - Log entry 5
2023-07-18 13:20:33 - Log entry 6

Output:

[get@me test]$ awk -F" " '$2 >= "09:00:00" && $2 <= "12:00:00"' server.log
2023-07-18 09:10:15 - Log entry 2
2023-07-18 10:30:47 - Log entry 3
2023-07-18 11:45:02 - Log entry 4
3
  • 1
    It's not known that time will start from 9:00:00 and it's not given that there will be time like 12:00:00. so how'd the regex change?
    – achhainsan
    Jul 19, 2023 at 3:30
  • sorry I'm quite puzzled by your followup question. The solution above shows logs that are timestamped between 09:00:00 and up to and including 12:00:00. These variables are time in 24 hr format as you described in your question - you adjust these two bounds depending on what time range interests you. What exactly do you mean by it's not known that time will start from 9:00:00 and it's not given that there will be time like 12:00:00.
    – varsock
    Jul 19, 2023 at 19:56
  • I've updated how the logs look like. Take a look.
    – achhainsan
    Aug 7, 2023 at 3:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .