Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am running a daemon process that will have to run indefinitely (a 'service' so to speak) and wish to log its output. A simple solution like:

./long-running-process > log.out &

... fails as the file log.out:

  • soon exceeds the size that I can easily handle with a text editor like emacs or vi
  • runs the risk of depleting free file system space.

To keep the size of the log file manageable I could use the split bash command:

./long-running-process | split -l 30000

This solution keeps the log files it creates manageable in size however it can run out of suffixes (split: output file suffixes exhausted) or, if the suffix space is huge, it may also deplete file system space.

So I am looking at a solution that will generate a number of log files, with each log file being manageable in size, and will also rotate amongst them so that there is a ceiling on the total disk space that will be claimed.

Is there such a solution available or do I have to implement it at the application level?

share|improve this question
    
logrotate in another word ? –  Kiwy Feb 17 at 13:28

3 Answers 3

up vote 2 down vote accepted

To cut the logs

Apache project has a useful command rotatelogs designed to rotate input recieved via an input pipe Read about rotatelogs

Then there is also the cronolog better time handling. Cronolog website

But if you are also rotating then it's worth considering logrotate, but logrotate will need a mechanism to trigger a new logfile, (send a signal, restart the program, ...). This is where rotatelogs/cronolog might come in, if you are logging stdout and do not want to restart the process.

share|improve this answer
    
rotatelogs satisfies one of my requirements (generate log files that are manageable in size), but it too can fill the file system unless I combine rotatelogs with another process that automatically deletes log files that are older than, say, 3 days. –  Marcus Junius Brutus Feb 17 at 14:09
    
OK so I guess rotatelogs with a size limit and a chronological timestamp coupled with a cronjob to delete old log files is exactly what I need since I won't have to worry about restarting processes, I am just logging stdout and expect rotatelogs to transparently switch to new files as I've seen it does. –  Marcus Junius Brutus Feb 17 at 14:51

Most modern Linux distros include a tool called logrotate which the OS then uses to maintain the /var/log directory. You can use it too. It is kicked off via cron, so if you want the logs rotated with a certain frequency then you need to setup a cronjob that runs atleast that frequently.

Examples

This will rotate the 2 files access.log & error.log, keeping at most 5 (current + 4 rotations). After relocating the current log file, killall -HUP httpd sends a "Hang Up" signal to the running daemon to trigger the creation of a new log file to start logging from this point on to the original named access.log and error.log files. This one will also rotate the log files if their size exceeds 100k.

   "/var/log/httpd/access.log" /var/log/httpd/error.log {
       rotate 5
       mail www@my.org
       size 100k
       sharedscripts
       postrotate
           /usr/bin/killall -HUP httpd
       endscript
   }

This one will rotate the log files under the directory /var/log/news/* montly, keeping 2 (current + 1). This set of rules will keep the logs in their original state, rather they will not be compressed (.gz) which is the default behavior.

   /var/log/news/* {
       monthly
       rotate 2
       olddir /var/log/news/old
       missingok
       postrotate
           kill -HUP `cat /var/run/inn.pid`
       endscript
       nocompress
   }

Do I have to send kill -HUP?

No this is not mandatory, only if your application requires it. This is what triggers your application to stop writing to the current log file (after it's been renamed from say access.log to access.log.1) and begin logging again to the original name, access.log.

/var/log/atop/atop.log {
    missingok
    weekly
    rotate 4 
    notifempty
    create 0600 root root
}

References

share|improve this answer
    
Thanks but unless I 've missed something I prefer the solution with rotatelogs and a cronjob to delete old log files (which is the only part that rotatelogs doesn't provide out of the box). Seems like less configuration is needed (only a cronjob), I don't have to worry about sending signals or restarting processes (why should I?) and my scripts (except the cronjob) reside in the project's file system hierarchy meaning they are under git as well. –  Marcus Junius Brutus Feb 17 at 14:55
    
@MarcusJuniusBrutus - I showed 2 examples for triggering the recreation of the logs, but if your daemon is written correctly you don't need to do that either. The triggering is needed b/c when you move a file to a new name the filedescriptor is still the same, so the daemon will continue writing to the new name. The signal triggers the app to stop doing this and start writing to a new log file with the original name. There is no escaping this behavior, it's how FD's work. BTW, a HUP is not a restart, 2 completely different things! –  slm Feb 17 at 15:00

for completeness sake i'd also like to mention the copytruncate option for logrotate:

   copytruncate
          Truncate the original log file to zero size in place after
          creating a copy, instead of moving the old log file and
          optionally creating a new one.  It can be used when some program
          cannot be told to close its logfile and thus might continue
          writing (appending) to the previous log file forever.
         Note that there  is  a  very small  time  slice  between  copying
          the  file  and truncating it, so some logging data might be lost.
          When this option is used, the *create* option will have no effect,
          as the old log file stays in place.
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.