Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I would like monitor a large log file (close to 1 GB) for errors. I want this to be close to real time (few seconds delay is fine). My plan is to use tail -f | grep. Is there any performance issues with using such a method when running it over a long time, say from zero bytes to 1 GB? Are there any standards practices used for such a monitoring. Note that I would like to do this using standard unix commands available on Solaris 10.

If that is possible, my file even rolls over and I have one more problem to sort out :). using tail -F (-follow=name) is not an option for me because -F is not supported in the server I want to run this on. My plan is to use a script which will start this tail and poll on to find if the file is rolled over. If yes, then kill the tail and restart it. Any better approach?

share|improve this question
    
You mean "kill the tail", right? –  Stéphane Gimenez Sep 7 '11 at 23:48
    
Yes, "kill tail", not find. Thanks, edited the question –  Manoj N V Sep 8 '11 at 4:58
    
If a 2 GB sized of file does not contain a new line character, how does tail work? –  user16336 Mar 7 '12 at 8:55
add comment

3 Answers

If it's invoked on a regular file (as opposed to a pipe), both GNU tail and OpenBSD tail (unless called with -n +N) seek to the end of the file, then works backwards to find the line where it should start printing. I don't know if Solaris does the same, but it's a reasonable approach, so I expect most unices to do the same. Therefore the size of the file is irrelevant for performance.

share|improve this answer
add comment

On my linux system (GNU coreutils 8.12), I was able to check (using strace) that tail -f¹ uses the lseek system call to skip over most of the file quickly:

lseek(3, 0, SEEK_CUR)                   = 0
lseek(3, 0, SEEK_END)                   = 194086
lseek(3, 188416, SEEK_SET)              = 188416

This means that the size of the tracked file should not matter in anyway.

Maybe you can check if the same applies on your system. (Obviously, it should be the case.)


1. I also tried disabling inotify support with the undocumented ---disable-inotify, just in case.

share|improve this answer
    
Real Men read the source (: –  Gilles Sep 8 '11 at 0:14
    
@Gilles. I can't teach the OP (or the reader) how to read the source if he doesn't know already. A lot easier to tell him to use strace ;) –  Stéphane Gimenez Sep 8 '11 at 0:20
    
Actually, on a system where tail -F is not supported, chances are that strace is not available… –  Stéphane Gimenez Sep 8 '11 at 0:23
    
truss is the corresponding utility on Solaris. –  Gilles Sep 8 '11 at 0:28
    
and it shows similar seek calls. llseek(0, 0, SEEK_CUR) = 0, llseek(0, 0xFFFFFFFFFFF5FFF6, SEEK_END) = 7923269 –  jlliagre Sep 8 '11 at 0:49
add comment

I do this every day. I generally scan a dozen or so logs on our test and production servers using tail -f logs/*.{log,err,out}. The initial load is a bit much (depending on the number of files globbed), but after that, streaming is real-time.

Instead of sending to grep, I use the exec functionality in screen since I want to generally see all the output (for full tracebacks and messages related to the issue). For example,

!:sed -n s/.*Exception.*/\007/p

To cause the terminal to beep (or flash) whenever the word Exception is found.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.