How to tail a log file by time?

Question

Say my log file(log.txt) is something like

2014-01-01 22:30:30 something happened....
2014-01-01 22:30:31 something happened....
2014-01-01 22:30:41 something happened....

I want to tail this file to show last hour's log, and keep tailing..

i.e.

tail <some magic to specify last 1 hour> -f log.txt

Then the output is

2014-01-01 21:30:41 something happened....
...
2014-01-01 22:30:30 something happened....
2014-01-01 22:30:31 something happened....
2014-01-01 22:30:41 something happened....

Is there a tool to do this?

Answer 1

You can just use combination of grep and tail in oneliner.

grep "2014-01-01 21:" log.txt; tail -f log.txt

It will print everything from that hour, and keep tailing.

or you can also use awk to print everything from start of an certain hour to end of file, and keep tailing after it, this will allow you to tail last few hours if needed.

awk '/2014-01-01 21:/' log.txt; tail -f log.txt

Answer 2

onehourago=$(date --date='1 hours ago' +"%b%e %H:%M:%S") 
echo $onehourago | cat /var/log/auth.log - | sort | sed "1,/$onehourago/d"

Intermediate output without the sed-command:

Aug  7 00:00:03 thinkpux CRON[25475]: pam_unix(cron:session): session closed for user stefan
Aug  7 00:17:01 thinkpux CRON[25504]: pam_unix(cron:session): session closed for user root
Aug  7 00:17:01 thinkpux CRON[25504]: pam_unix(cron:session): session opened for user root by (uid=0)
Aug 7 00:19:33
Aug  7 01:00:02 thinkpux CRON[25652]: pam_unix(cron:session): session opened for user stefan by (uid=0)
Aug  7 01:00:03 thinkpux CRON[25652]: pam_unix(cron:session): session closed for user stefan
Aug  7 01:17:01 thinkpux CRON[25885]: pam_unix(cron:session): session closed for user root
Aug  7 01:17:01 thinkpux CRON[25885]: pam_unix(cron:session): session opened for user root by (uid=0)

Output:

Aug  7 01:00:02 thinkpux CRON[25652]: pam_unix(cron:session): session opened for user stefan by (uid=0)
Aug  7 01:00:03 thinkpux CRON[25652]: pam_unix(cron:session): session closed for user stefan
Aug  7 01:17:01 thinkpux CRON[25885]: pam_unix(cron:session): session closed for user root
Aug  7 01:17:01 thinkpux CRON[25885]: pam_unix(cron:session): session opened for user root by (uid=0)

Note: First I had build the date with +"%b %e %H:%M:%S" (note the blank between %b and %e, but echo compresses two blanks into one. Searching by sed seems to look for two blanks and doesn't find the expression, so it is somehow complicated to distinguish whether the day of month has two or one digit. However, sort seems agnostic about the superflous/missing blank. Maybe there is a bash-switch to prevent compression? However - your dateformat is different and so you will not be affected, but I had to test this approach somehow, and learners might step into the same trap.

So the overall Idea is, to get the date/time of 1 hour ago, format it as in the logfile, add the pure date to the logfile and sort that mix, then delete with sed anything up to the pure date.

Not funny to remember and type, but you may put it into a script and/or function.

Answer 3

You could use a combination of tail -f and perl:

# Tail with timestamp
tail -f log.txt | perl -pe '$_ = localtime.": $_"'

This will provide an output on the terminal something like this:

You can also save this stdout in a file as following:

tail -f log.txt | perl -pe '$_ = localtime.": $_"' >> timestamped_log.txt

Hopefully, this provides a solution

Answer 4

• Problem with grep or awk is, that you always need to read the whole file and check every line.
• Problem with tail is, you don't know how many lines you need to get.

So, try tac and awk and then tail -f with -n0.

tac file \
  | awk -F'[- :]' -v datefilter="$(date -d -1hour +%s)" 'mktime($1 OFS $2 OFS $3 OFS $4 OFS $5 OFS $6) < datefilter {exit} 1' \
  | tac
tail -f -n0

(Must be said, that if there are new lines coming in between the two commands you might lose them).