2

I am trying to go through an IPA host's log file to grab the last five minutes of content and get updates on things like new user creations or DNS changes. I want to cron this so every five minutes it will check on the previous five minutes, go through and match on some strings and email those results. I do not know how to easily compare the strings of the type presented in the log:

20200114184803 = 2020-01-14 | 18:48:03

time: 20200114184803
Stuff
Stuff
Stuff

time: 20200114184804
Stuff
Stuff
Stuff

time: 20200114184811
Stuff
Stuff
Stuff
  • It's easier to run on 10 minute intervals, as you can simply grep the time to the 10 minute place 20200114184 – rtaft Jan 14 at 19:13
  • So if you get run at 2002-01-14 15:30:01, you want to find all entries at or after 2002-01-14 15:25:00? Two problems with that: (a) It's not just grep, you need to do date arithmetic. Midnight and clock change are much fun. (b) If the first event in the time interval is 15:27:55, your grep has to be like 15:2[56789]. I found the easiest way to do this it not to cron separate runs: instead, remember where you read up to last time, tail -n +nn to pick up fresh lines, wc -l on the new data, remember the new total length, sleep 300, loop. – Paul_Pedant Jan 14 at 19:27
0

I broke it out into a couple variables but you can probably next or pipe it a little better:

fma=`date --date='5 minutes ago' +"%Y%m%d%H%M%S"`
line_num=`tac file.txt | awk -v five_min_ago=$fma\
      '(NF > 1 &&  $2 < five_min_ago ){print NR; exit 1}'`
tail -$line_num file.txt

Explanation:

  1. fma=... says store in shell variable fma the date five minutes ago in the format year, month, day, hour, min, second, which appears to be what your log does.
  2. line_num=... says read the file backwards and pipe that to awk. In the awk program, save the bash variable $fma as five_minutes_ago, then if the number of fields is more than just one and the log time is less than the target time, print out the number of rows we've seen so far (including ones that didn't qualify) and exit. Store the result as shell variable line_num.
  3. Finally, use tail to get the last line_num lines from file.txt

Note: You will have to adjust the number of fields check since you probably have more than "stuff" there, but this should get you most of the way.

| improve this answer | |
  • Sorry this does not seem to return the expected output even with tailoring the input to match what I gave in the original example. – Kahn Jan 14 at 20:38
  • Even if this worked with the sample data, it would not work in a real environment. A log entry could be made between the calculation of the line number and the tail call, which would result in losing logs you want and gaining logs from the next window. – rtaft Jan 15 at 15:11
  • @rtaft in general script answers here and elsewhere don't account for race conditions. Here it would be fairly easy to cp the target file first to get a snapshot and consider that the canonical. – user1794469 Jan 15 at 16:06
0

I've solved the problem with awk in my script:

ONE_HOUR_AGO="$(date -d '1 hour ago' +%Y%m%d%H)"
NOW="$(date +%Y%m%d%H)"
awk '/time: '$NOW'/,/time: '$ONE_HOUR_AGO'/' log_file.txt

This has fixed my issues with these timestamps and I'm able to parse the log file.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.