Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm managing a server which works as a remote log repository for huge number of files. Currently a single logrotate instance cannot keep up with this number of logs. What's the best way to accelerate log rotation, possibly running many instances in parallel?

Below is the addendum for the clarification:

  • Disk I/O is not the problem; also, I'm not losing any logs or entries.
  • System is a beast when compared to this job (it's a quad core Xeon with sufficient power) the bottleneck is at compression. One core tops up w/ 1.00 load and cannot finish its compression queue.
  • Logfiles are not copied. Only a daily backup is taken to another disk, but again is not a bottleneck.
  • I can group the logfiles by patterns. Files are not huge, only they are a lot. I just don't know how to parallelize the logrotate altough I have ideas but unable to find any resources for it. Actually, the core of the question is does logrotate spawn a child process for every stub in the config file or configuration file?
  • I'm a computer engineer and an experienced Linux user and developer. I also manage a cluster but somehow I'm new at "old school system management" issues. So you can go as technical / hardcore as you can get. I'll do my best to understand it.
share|improve this question
add comment

3 Answers

up vote 3 down vote accepted

You can run multiple instances of logrotate. They need to work on different sets of files and use different state-files.

You should use gzip (standard compression) instead of bzip. AFAIK you can pass some extra options to gzip - so you can tell it to use faster compression.

share|improve this answer
add comment

One thing that might help is not putting all the logs in the same directory - having a lot of files in the same directory can degrade performance a lot.

share|improve this answer
    
No, directories are not crowded in terms of logs. –  SilentStorm Jan 30 '12 at 19:44
add comment

Please tell us more about your requirements - it's hard to guess what limits your server:

  • disk i/o? You might want to spread out those logfiles over disks/filesystems
  • cpu - is it compressing those logs as they are rotated? You might want to use a filesystem with internal compression, and even hardware acceleration.
  • directory cache? see the answer by Chris Card - spread your files over directories, and, just as important, make those files differer in the early characters of the filename, or you will not see the benefits of the directory cache applied to them, which slows down lookups significantly.
  • if your server is meeting its limits - is it just so for log rotation, or is it already missing out on log entries?
  • Are you copying those logfiles around, or are you moving them (within the same filesystem)?
  • ...

If you design a structure for your logfiles, there's nothing that would keep you from parallelizing rotating them.

edit in reply to added requirements

As far as I know, logrotate does not parallelize in itself. Once you have set up a clear structure though, there are a few roads you can explore to parallelize it (or at least the parts bogging you down) manually:

  • The compress command itself: You can use a wrapper script to do the compression. The gzip spawned can run in the background while your script returns immediately - thus leaving the compress job running while logrotate merrily proceeds to the next log file. Two caveats:
    1. I'd rather be safe and switch on delaycompress to avoid still fiddling with the logfile while the process is being SIGHUP'ed or whatever it takes to start a new log (depending on the process).
    2. I'd also "nice" the compress job, to avoid having many parallel compress jobs compete for cpu time.
  • If you're feeling sophisticated (or end up with numerous parallel gzip jobs throttling down the loghost), you may later transition to writing a set of worker processes taking their files-to-compress from a list generated by the "compress command" script stated in your logrotate config.
  • You can have independent instances of logrotate running with a bit of planning:
    • each instance needs its own config file.
    • each instance needs its own set of directories to watch for logfiles. You can (and probably should) have separate crontab entries for them.
    • each instance needs its own statefile (configurable)
share|improve this answer
    
Except I do not think that logrotate can handle parallel runs, so he may have to roll his own. –  Alien Life Form Jan 30 '12 at 16:43
    
It has support for multiple state files. In fact I always find the overloaded logrotate process as a child of another logrotate process. I'm somewhat inclined to think logrotate parallelizes itself on config file or config stub level. –  SilentStorm Jan 30 '12 at 21:47
    
That's both correct. While logrotate itself is awfully sequential (for a reason, they want to avoid running into race conditions with the process writing the logfile, hence the requirement to use delaycompress if the compression part gets parallelized), some kind of parallelization can be achieved by having several instances of logrotate take care of independant sets of logfiles. –  Tatjana Heuser Jan 30 '12 at 22:56
    
Thanks for the details. –  SilentStorm Jan 31 '12 at 7:03
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.