2

I'm managing a server which works as a remote log repository for huge number of files. Currently a single logrotate instance cannot keep up with this number of logs. What's the best way to accelerate log rotation, possibly running many instances in parallel?

Below is the addendum for the clarification:

  • Disk I/O is not the problem; also, I'm not losing any logs or entries.
  • System is a beast when compared to this job (it's a quad core Xeon with sufficient power) the bottleneck is at compression. One core tops up w/ 1.00 load and cannot finish its compression queue.
  • Logfiles are not copied. Only a daily backup is taken to another disk, but again is not a bottleneck.
  • I can group the logfiles by patterns. Files are not huge, only they are a lot. I just don't know how to parallelize the logrotate altough I have ideas but unable to find any resources for it. Actually, the core of the question is _does logrotate spawn a child process for every stub in the config file or configuration file?

4 Answers 4

3

You can run multiple instances of logrotate. They need to work on different sets of files and use different state-files.

You should use gzip (standard compression) instead of bzip. AFAIK you can pass some extra options to gzip - so you can tell it to use faster compression.

3

Please tell us more about your requirements - it's hard to guess what limits your server:

  • disk i/o? You might want to spread out those logfiles over disks/filesystems
  • cpu - is it compressing those logs as they are rotated? You might want to use a filesystem with internal compression, and even hardware acceleration.
  • directory cache? see the answer by Chris Card - spread your files over directories, and, just as important, make those files differer in the early characters of the filename, or you will not see the benefits of the directory cache applied to them, which slows down lookups significantly.
  • if your server is meeting its limits - is it just so for log rotation, or is it already missing out on log entries?
  • Are you copying those logfiles around, or are you moving them (within the same filesystem)?
  • ...

If you design a structure for your logfiles, there's nothing that would keep you from parallelizing rotating them.

edit in reply to added requirements

As far as I know, logrotate does not parallelize in itself. Once you have set up a clear structure though, there are a few roads you can explore to parallelize it (or at least the parts bogging you down) manually:

  • The compress command itself: You can use a wrapper script to do the compression. The gzip spawned can run in the background while your script returns immediately - thus leaving the compress job running while logrotate merrily proceeds to the next log file. Two caveats:
    1. I'd rather be safe and switch on delaycompress to avoid still fiddling with the logfile while the process is being SIGHUP'ed or whatever it takes to start a new log (depending on the process).
    2. I'd also "nice" the compress job, to avoid having many parallel compress jobs compete for cpu time.
  • If you're feeling sophisticated (or end up with numerous parallel gzip jobs throttling down the loghost), you may later transition to writing a set of worker processes taking their files-to-compress from a list generated by the "compress command" script stated in your logrotate config.
  • You can have independent instances of logrotate running with a bit of planning:
    • each instance needs its own config file.
    • each instance needs its own set of directories to watch for logfiles. You can (and probably should) have separate crontab entries for them.
    • each instance needs its own statefile (configurable)
3
  • Except I do not think that logrotate can handle parallel runs, so he may have to roll his own. Jan 30, 2012 at 16:43
  • It has support for multiple state files. In fact I always find the overloaded logrotate process as a child of another logrotate process. I'm somewhat inclined to think logrotate parallelizes itself on config file or config stub level.
    – bayindirh
    Jan 30, 2012 at 21:47
  • That's both correct. While logrotate itself is awfully sequential (for a reason, they want to avoid running into race conditions with the process writing the logfile, hence the requirement to use delaycompress if the compression part gets parallelized), some kind of parallelization can be achieved by having several instances of logrotate take care of independant sets of logfiles. Jan 30, 2012 at 22:56
1

One thing that might help is not putting all the logs in the same directory - having a lot of files in the same directory can degrade performance a lot.

1
  • No, directories are not crowded in terms of logs.
    – bayindirh
    Jan 30, 2012 at 19:44
1

If your hosts have many cpu cores but your default compression utilities are single threaded and hitting a bottleneck on a single core, you can leverage more compression throughput with those other cores to complete compression tasks an order of magnitude faster. Examples are pigz in place of gzip and pbzip2 in place of bzip2.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .