Advanced logrotate on huge number of files when single instance of logrotate cannot keep up

Question

I'm managing a server which works as a remote log repository for huge number of files. Currently a single logrotate instance cannot keep up with this number of logs. What's the best way to accelerate log rotation, possibly running many instances in parallel?

Below is the addendum for the clarification:

• Disk I/O is not the problem; also, I'm not losing any logs or entries.
• System is a beast when compared to this job (it's a quad core Xeon with sufficient power) the bottleneck is at compression. One core tops up w/ 1.00 load and cannot finish its compression queue.
• Logfiles are not copied. Only a daily backup is taken to another disk, but again is not a bottleneck.
• I can group the logfiles by patterns. Files are not huge, only they are a lot. I just don't know how to parallelize the logrotate altough I have ideas but unable to find any resources for it. Actually, the core of the question is does logrotate spawn a child process for every stub in the config file or configuration file?
• I'm a computer engineer and an experienced Linux user and developer. I also manage a cluster but somehow I'm new at "old school system management" issues. So you can go as technical / hardcore as you can get. I'll do my best to understand it.

Answer 1

You can run multiple instances of logrotate. They need to work on different sets of files and use different state-files.

You should use gzip (standard compression) instead of bzip. AFAIK you can pass some extra options to gzip - so you can tell it to use faster compression.

Answer 2

Please tell us more about your requirements - it's hard to guess what limits your server:

• disk i/o? You might want to spread out those logfiles over disks/filesystems
• cpu - is it compressing those logs as they are rotated? You might want to use a filesystem with internal compression, and even hardware acceleration.
• directory cache? see the answer by Chris Card - spread your files over directories, and, just as important, make those files differer in the early characters of the filename, or you will not see the benefits of the directory cache applied to them, which slows down lookups significantly.
• if your server is meeting its limits - is it just so for log rotation, or is it already missing out on log entries?
• Are you copying those logfiles around, or are you moving them (within the same filesystem)?
• ...

If you design a structure for your logfiles, there's nothing that would keep you from parallelizing rotating them.

edit in reply to added requirements

As far as I know, logrotate does not parallelize in itself. Once you have set up a clear structure though, there are a few roads you can explore to parallelize it (or at least the parts bogging you down) manually:

• The compress command itself: You can use a wrapper script to do the compression. The gzip spawned can run in the background while your script returns immediately - thus leaving the compress job running while logrotate merrily proceeds to the next log file. Two caveats:
  1. I'd rather be safe and switch on delaycompress to avoid still fiddling with the logfile while the process is being SIGHUP'ed or whatever it takes to start a new log (depending on the process).
  2. I'd also "nice" the compress job, to avoid having many parallel compress jobs compete for cpu time.
• If you're feeling sophisticated (or end up with numerous parallel gzip jobs throttling down the loghost), you may later transition to writing a set of worker processes taking their files-to-compress from a list generated by the "compress command" script stated in your logrotate config.
• You can have independent instances of logrotate running with a bit of planning:
  • each instance needs its own config file.
  • each instance needs its own set of directories to watch for logfiles. You can (and probably should) have separate crontab entries for them.
  • each instance needs its own statefile (configurable)

Answer 3

One thing that might help is not putting all the logs in the same directory - having a lot of files in the same directory can degrade performance a lot.