Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm looking for a fairly simple way (no development involved -- I could write this in Python but I'm hoping there's something already out there).

I have a log file (in my case written by rsyslogd). For analytics purposes I want to read it every 1 minute and calculate metrics for the last minute e.g. how many page hits my http server had. My 2 requirements:

1) I only want to look at the lines that were added since I last read the file. (I only need the last minute or so, and the file is too large to re-read and filter every minute).

2) Once a day the file gets logrorate'd. The first time after log rotation I want all of the lines from the previous file I haven't yet read, plus all of the lines from the new file.

I assume I'm now the only one with such requirements -- what do others do?

share|improve this question
1  
Just use logtail. linux.die.net/man/8/logtail –  Tim Kennedy Aug 15 '13 at 5:45
    
@Tim -- nice, but when the log gets rotated I need to print the last lines from the previous log and then all the lines in the current log. –  Nitzan Shaked Aug 15 '13 at 7:38
add comment

1 Answer 1

up vote 2 down vote accepted

Assuming your-filter reads its data from stdin:

while your-filter; do
  sleep 60
done < file.log

That assumes your-filter just reads the data and doesn't attempt to lseek in it for instance.

Now, to address the log rotation issue, if on Linux (where, contrary to most other systems, /dev/fd/n are symlinks to the actual files), with ksh, bash, zsh, dash, yash (most POSIX shells except the most pedantically POSIX ones like posh as -ef is not POSIX):

while your-filter; do
  if [ file.log -ef /dev/stdin ]; then
    sleep 60
  else
    exec < file.log
  fi
done < file.log

Upon the log rotation, that would call your-filter twice, if you'd rather it being called once with the concatenation of the old and new:

while 
  if [ file.log -ef /dev/stdin ]; then
    your-filter
  else
    exec 3<&0 < file.log
    (cat <&3; cat) | your-filter &&
      exec 3<&-
  fi
do
  sleep 60
done < file.log

Now upon the log rotation, there may be a time when the old file.log has been renamed, but the new file.log not created yet, in which case the above will fail if it does the exec < file.log at that very moment. Then you could fix that with:

while 
  if [ file.log -ef /dev/stdin ] || ! command exec 3< file.log; then
    your-filter
  else
    (cat; cat <&3) | your-filter &&
      exec <&3 3<&-
  fi
do
  sleep 60
done < file.log

So it carries on reading the old file until the new one shows up.

command is needed to avoid exec to cause the shell to exit when it fails (as POSIX requires). It's not needed with zsh or bash when not in sh mode.

Now, we sleep for 60 seconds in the loop, and your-filter might take a few seconds to run. If it's important for your-filter to be run every minute on average, with ksh, bash or zsh, you could change it to:

t=$SECONDS
while 
  if [ file.log -ef /dev/stdin ] || ! command exec 3< file.log; then
    your-filter
  else
    (cat; cat <&3) | your-filter &&
      exec <&3 3<&-
  fi
do
  t=$(($t + 60))
  sleep "$((t - SECONDS))"
done < file.log

With ksh93 and zsh, and provided your sleep accepts floating point arguments, you could run typeset -F SECONDS.

share|improve this answer
    
that's a fantastic answer, thank you. I wish I could upvote more than once. Not only is it a complete solution to what I asked, I also like how it is structured. Finally -- your answer made me go and read all the tldp.org chapter 20 (redirection, exec, block redirection) again, and I feel somewhat smarter now :-) Thanks. –  Nitzan Shaked Aug 14 '13 at 17:15
    
Why not tail -F | while sleep 60; …? –  Gilles Aug 14 '13 at 21:35
    
@Gilles, because your-filter would block as it would never see the end-of-file. –  Stéphane Chazelas Aug 14 '13 at 21:37
    
True, you'd need a splitting pass first. Something built on tail -F | while true; do timeout 60 cat | your-filter; done. I get the impression that you're reimplementing tail -F. (Portability could be a reason for that of course.) –  Gilles Aug 14 '13 at 21:41
    
@Gilles -- cute. The timeout is only for cat, to make it stop reading it's stdin and make your-filter seen EOF, correct? Btw, I think this won't work if logrotate moves file.log using mv instead of copying it over and creating a new file. tail won't see that the file has moved, correct? –  Nitzan Shaked Aug 15 '13 at 5:03
show 3 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.