0

I am trying to filter a log file based on the following:

date range of an error (yyyy-mm-dd)

timestamp range of an error (01:00:00 - 00:00:00)

keywords (previousFireTime, nextFireTime)

I've been trying to grep the log file, however, I am unable to get the data that I need with using grep. Would I have to use a combination of awk and grep or awk and sed to get the information I need? Or would there be a better more efficient route to go by to filter a log file?

Edit: Example log output

2018-06-06 10:46:43,708 INFO [stdout] (AsyncAppender-Worker-STDOUT) INFO 
[erFactoryBean_Worker-9] [c.c.c.s.i.d.ResendJob] Executing Quartz scheduled 
job: JobExecutionContext: trigger: 'ResendJob.trigger_ResendJob job: 
DEFAULT.ResendJob fireTime: 'Wed Jun 06 10:46:43 UTC 2018 scheduledFireTime: 
Wed Jun 06 10:46:43 UTC 2018 previousFireTime: 'Wed Jun 06 10:45:43 UTC 2018 
nextFireTime: Wed Jun 06 10:47:43 UTC 2018 isRecovering: false refireCount: 0
4
  • 1
    An actual example of the log file along with exactly what data you'd like to extract from it would be helpful. Also note that awk usually does not need to be combined with either of grep or sed as it does most of what these other utilities do already.
    – Kusalananda
    Jun 6, 2018 at 15:23
  • Hi, sorry about that an example of the log output would be as follows: 2018-06-06 10:46:43,708 INFO [stdout] (AsyncAppender-Worker-STDOUT) INFO [erFactoryBean_Worker-9] [c.c.c.s.i.d.ResendResponsesJob] Executing Quartz scheduled job: JobExecutionContext: trigger: 'ResendResponsesJob.trigger_ResendResponsesJob job: DEFAULT.ResendResponsesJob fireTime: 'Wed Jun 06 10:46:43 UTC 2018 scheduledFireTime: Wed Jun 06 10:46:43 UTC 2018 previousFireTime: 'Wed Jun 06 10:45:43 UTC 2018 nextFireTime: Wed Jun 06 10:47:43 UTC 2018 isRecovering: false refireCount: 0
    – Lemn
    Jun 6, 2018 at 16:34
  • I would then like to filter that based on a date and time range along with keywords like 'resendresponsesjob'
    – Lemn
    Jun 6, 2018 at 16:34
  • 1
    Don't add clarifications in comments. Edit the question instead.
    – Kusalananda
    Jun 6, 2018 at 17:16

2 Answers 2

2

awk is probably all you need here as it can do regexp matching, split lines into fields and do string comparisons (which works for date comparison as long as you use YYYY-MM-DD HH:MM:SS timestamps and there's no DST change).

If the date is in the first field and time in second:

awk -v date=1 -v time=2 '
  $date > "2018-05-24" && $time < "12:00:00" && /some text/'

The GNU awk implementation of awk has date parsing and formatting extensions which lets you do more advanced stuff like:

gawk -v date=1 -v time=2 '
  function parse_time(t) {
    gsub(/[:-]/, " ", t)
    return mktime(t)
  }
  BEGIN {
    start = parse_time("2018-01-01 08:00")
    end = systime() - 86400 # yesterday, same time
  }
  {t = parse_time($date" "$time)}
  t >= start && t <= end && /some test/'
1
  • Thank you for the explanation, I appreciate it! Would I be able to add more information with awk to filter based on keywords as well? Or would it only work based on time and date range? Example for keyword: nextRunTime, previousRunTime
    – Lemn
    Jun 6, 2018 at 16:35
1

grep filters regular expressions. It is very good at filtering lines containing one particular keyword, but a date range is hard to specify using regular expressions. For example, to get errors between Jan 1st, 20:00 and Jan 3rd, 2:00, you have to accept all times for Jan 2nd, but only evening for Jan 1st and only early morning for Jan 3rd. You can't separate time of day and date, for example.

It is much simpler to use a more expressive tool that can natively compare dates. Perl is a popular language to do this kind of things, and Python is a good alternative.

Here's an example in Python:

import re
import time

f = open('/var/log/syslog')
line = f.readline()
while line:
    # Get the date at the beginning of line with a regex
    m = re.match(r'^([^\s]+\s+[^\s]+\s+[^\s]+)\s+', line)
    # Parse the date
    date = time.strptime(m.group(1), '%b %d %H:%M:%S')
    # Compare with a given date
    if date > time.strptime('Jun 6 14:00:00', '%b %d %H:%M:%S'):
        print(line, end='')

    # Read next line
    line = f.readline()
1
  • Gotcha, thank you for the clarification! I was thinking of using a script for this. Leaning more towards going down the script route at this point, I think.
    – Lemn
    Jun 6, 2018 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.