solution to rotate log files

Question

I am running a daemon process that will have to run indefinitely (a 'service' so to speak) and wish to log its output. A simple solution like:

./long-running-process > log.out &

... fails as the file log.out:

• soon exceeds the size that I can easily handle with a text editor like emacs or vi
• runs the risk of depleting free file system space.

To keep the size of the log file manageable I could use the split bash command:

./long-running-process | split -l 30000

This solution keeps the log files it creates manageable in size however it can run out of suffixes (split: output file suffixes exhausted) or, if the suffix space is huge, it may also deplete file system space.

So I am looking at a solution that will generate a number of log files, with each log file being manageable in size, and will also rotate amongst them so that there is a ceiling on the total disk space that will be claimed.

Is there such a solution available or do I have to implement it at the application level?

Answer 1

To cut the logs

Apache project has a useful command rotatelogs designed to rotate input recieved via an input pipe Read about rotatelogs

Then there is also the cronolog better time handling. Cronolog website

But if you are also rotating then it's worth considering logrotate, but logrotate will need a mechanism to trigger a new logfile, (send a signal, restart the program, ...). This is where rotatelogs/cronolog might come in, if you are logging stdout and do not want to restart the process.

Answer 2

Most modern Linux distros include a tool called logrotate which the OS then uses to maintain the /var/log directory. You can use it too. It is kicked off via cron, so if you want the logs rotated with a certain frequency then you need to setup a cronjob that runs atleast that frequently.

Examples

This will rotate the 2 files access.log & error.log, keeping at most 5 (current + 4 rotations). After relocating the current log file, killall -HUP httpd sends a "Hang Up" signal to the running daemon to trigger the creation of a new log file to start logging from this point on to the original named access.log and error.log files. This one will also rotate the log files if their size exceeds 100k.

   "/var/log/httpd/access.log" /var/log/httpd/error.log {
       rotate 5
       mail [email protected]
       size 100k
       sharedscripts
       postrotate
           /usr/bin/killall -HUP httpd
       endscript
   }

This one will rotate the log files under the directory /var/log/news/* montly, keeping 2 (current + 1). This set of rules will keep the logs in their original state, rather they will not be compressed (.gz) which is the default behavior.

   /var/log/news/* {
       monthly
       rotate 2
       olddir /var/log/news/old
       missingok
       postrotate
           kill -HUP `cat /var/run/inn.pid`
       endscript
       nocompress
   }

Do I have to send kill -HUP?

No this is not mandatory, only if your application requires it. This is what triggers your application to stop writing to the current log file (after it's been renamed from say access.log to access.log.1) and begin logging again to the original name, access.log.

/var/log/atop/atop.log {
    missingok
    weekly
    rotate 4 
    notifempty
    create 0600 root root
}

References

• HowTo: The Ultimate Logrotate Command Tutorial with 10 Examples

Answer 3

for completeness sake i'd also like to mention the copytruncate option for logrotate:

   copytruncate
          Truncate the original log file to zero size in place after
          creating a copy, instead of moving the old log file and
          optionally creating a new one.  It can be used when some program
          cannot be told to close its logfile and thus might continue
          writing (appending) to the previous log file forever.
         Note that there  is  a  very small  time  slice  between  copying
          the  file  and truncating it, so some logging data might be lost.
          When this option is used, the *create* option will have no effect,
          as the old log file stays in place.