6

I am running a daemon process that will have to run indefinitely (a 'service' so to speak) and wish to log its output. A simple solution like:

./long-running-process > log.out &

... fails as the file log.out:

  • soon exceeds the size that I can easily handle with a text editor like emacs or vi
  • runs the risk of depleting free file system space.

To keep the size of the log file manageable I could use the split bash command:

./long-running-process | split -l 30000

This solution keeps the log files it creates manageable in size however it can run out of suffixes (split: output file suffixes exhausted) or, if the suffix space is huge, it may also deplete file system space.

So I am looking at a solution that will generate a number of log files, with each log file being manageable in size, and will also rotate amongst them so that there is a ceiling on the total disk space that will be claimed.

Is there such a solution available or do I have to implement it at the application level?

  • logrotate in another word ? – Kiwy Feb 17 '14 at 13:28
7

To cut the logs

Apache project has a useful command rotatelogs designed to rotate input recieved via an input pipe Read about rotatelogs

Then there is also the cronolog better time handling. Cronolog website

But if you are also rotating then it's worth considering logrotate, but logrotate will need a mechanism to trigger a new logfile, (send a signal, restart the program, ...). This is where rotatelogs/cronolog might come in, if you are logging stdout and do not want to restart the process.

  • rotatelogs satisfies one of my requirements (generate log files that are manageable in size), but it too can fill the file system unless I combine rotatelogs with another process that automatically deletes log files that are older than, say, 3 days. – Marcus Junius Brutus Feb 17 '14 at 14:09
  • OK so I guess rotatelogs with a size limit and a chronological timestamp coupled with a cronjob to delete old log files is exactly what I need since I won't have to worry about restarting processes, I am just logging stdout and expect rotatelogs to transparently switch to new files as I've seen it does. – Marcus Junius Brutus Feb 17 '14 at 14:51
5

Most modern Linux distros include a tool called logrotate which the OS then uses to maintain the /var/log directory. You can use it too. It is kicked off via cron, so if you want the logs rotated with a certain frequency then you need to setup a cronjob that runs atleast that frequently.

Examples

This will rotate the 2 files access.log & error.log, keeping at most 5 (current + 4 rotations). After relocating the current log file, killall -HUP httpd sends a "Hang Up" signal to the running daemon to trigger the creation of a new log file to start logging from this point on to the original named access.log and error.log files. This one will also rotate the log files if their size exceeds 100k.

   "/var/log/httpd/access.log" /var/log/httpd/error.log {
       rotate 5
       mail www@my.org
       size 100k
       sharedscripts
       postrotate
           /usr/bin/killall -HUP httpd
       endscript
   }

This one will rotate the log files under the directory /var/log/news/* montly, keeping 2 (current + 1). This set of rules will keep the logs in their original state, rather they will not be compressed (.gz) which is the default behavior.

   /var/log/news/* {
       monthly
       rotate 2
       olddir /var/log/news/old
       missingok
       postrotate
           kill -HUP `cat /var/run/inn.pid`
       endscript
       nocompress
   }

Do I have to send kill -HUP?

No this is not mandatory, only if your application requires it. This is what triggers your application to stop writing to the current log file (after it's been renamed from say access.log to access.log.1) and begin logging again to the original name, access.log.

/var/log/atop/atop.log {
    missingok
    weekly
    rotate 4 
    notifempty
    create 0600 root root
}

References

  • Thanks but unless I 've missed something I prefer the solution with rotatelogs and a cronjob to delete old log files (which is the only part that rotatelogs doesn't provide out of the box). Seems like less configuration is needed (only a cronjob), I don't have to worry about sending signals or restarting processes (why should I?) and my scripts (except the cronjob) reside in the project's file system hierarchy meaning they are under git as well. – Marcus Junius Brutus Feb 17 '14 at 14:55
  • @MarcusJuniusBrutus - I showed 2 examples for triggering the recreation of the logs, but if your daemon is written correctly you don't need to do that either. The triggering is needed b/c when you move a file to a new name the filedescriptor is still the same, so the daemon will continue writing to the new name. The signal triggers the app to stop doing this and start writing to a new log file with the original name. There is no escaping this behavior, it's how FD's work. BTW, a HUP is not a restart, 2 completely different things! – slm Feb 17 '14 at 15:00
  • It sounds like you're contradicting yourself. You're saying -HUP is not always mandatory, but then you say "There is no escaping this behaviour". Surely a daemon needs to know somehow when to start logging to the new file. – Phil_1984_ Jun 20 at 10:11
  • @Phil_1984_ - The requirement of explicitly calling a kill -HUP in the postrotate section. It comes down to implicitly relying on logrotate moving the file out of the way and the app picking up on this and starting a new file vs. having to explicitly trigger the creation of a new file after its been moved via sending the app a signal w/ a kill -HUP. – slm Jun 20 at 11:37
2

for completeness sake i'd also like to mention the copytruncate option for logrotate:

   copytruncate
          Truncate the original log file to zero size in place after
          creating a copy, instead of moving the old log file and
          optionally creating a new one.  It can be used when some program
          cannot be told to close its logfile and thus might continue
          writing (appending) to the previous log file forever.
         Note that there  is  a  very small  time  slice  between  copying
          the  file  and truncating it, so some logging data might be lost.
          When this option is used, the *create* option will have no effect,
          as the old log file stays in place.
  • copytruncate doesnt need any package install. So is an apt solution for logs other Apache's. – Vishnu Kumar Apr 21 '17 at 9:48
  • 1
    copytruncate does solve the issue of daemon signalling to open a new fd but introduces race conditions where log data may get "lost" (i.e. Never logged) during the rotation process. You can't copy all data and truncate a file in a single atomic POSIX operation. Normally this would be a deal breaker for anyone requiring log data consistency when writing to files. – Phil_1984_ Jun 20 at 10:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.