Periodically get new lines from file, possibly logrotated

Question

I'm looking for a fairly simple way (no development involved -- I could write this in Python but I'm hoping there's something already out there).

I have a log file (in my case written by rsyslogd). For analytics purposes I want to read it every 1 minute and calculate metrics for the last minute e.g. how many page hits my http server had. My 2 requirements:

1) I only want to look at the lines that were added since I last read the file. (I only need the last minute or so, and the file is too large to re-read and filter every minute).

2) Once a day the file gets logrorate'd. The first time after log rotation I want all of the lines from the previous file I haven't yet read, plus all of the lines from the new file.

I assume I'm now the only one with such requirements -- what do others do?

Answer 1

Assuming your-filter reads its data from stdin:

while your-filter; do
  sleep 60
done < file.log

That assumes your-filter just reads the data and doesn't attempt to lseek in it for instance.

Now, to address the log rotation issue, if on Linux (where, contrary to most other systems, /dev/fd/n are symlinks to the actual files), with ksh, bash, zsh, dash, yash (most POSIX shells except the most pedantically POSIX ones like posh as -ef is not POSIX):

while your-filter; do
  if [ file.log -ef /dev/stdin ]; then
    sleep 60
  else
    exec < file.log
  fi
done < file.log

Upon the log rotation, that would call your-filter twice, if you'd rather it being called once with the concatenation of the old and new:

while 
  if [ file.log -ef /dev/stdin ]; then
    your-filter
  else
    exec 3<&0 < file.log
    (cat <&3; cat) | your-filter &&
      exec 3<&-
  fi
do
  sleep 60
done < file.log

Now upon the log rotation, there may be a time when the old file.log has been renamed, but the new file.log not created yet, in which case the above will fail if it does the exec < file.log at that very moment. Then you could fix that with:

while 
  if [ file.log -ef /dev/stdin ] || ! command exec 3< file.log; then
    your-filter
  else
    (cat; cat <&3) | your-filter &&
      exec <&3 3<&-
  fi
do
  sleep 60
done < file.log

So it carries on reading the old file until the new one shows up.

command is needed to avoid exec to cause the shell to exit when it fails (as POSIX requires). It's not needed with zsh or bash when not in sh mode.

Now, we sleep for 60 seconds in the loop, and your-filter might take a few seconds to run. If it's important for your-filter to be run every minute on average, with ksh, bash or zsh, you could change it to:

t=$SECONDS
while 
  if [ file.log -ef /dev/stdin ] || ! command exec 3< file.log; then
    your-filter
  else
    (cat; cat <&3) | your-filter &&
      exec <&3 3<&-
  fi
do
  t=$(($t + 60))
  sleep "$((t - SECONDS))"
done < file.log

With ksh93 and zsh, and provided your sleep accepts floating point arguments, you could run typeset -F SECONDS.