Please tell us more about your requirements - it's hard to guess what limits your server:
- disk i/o? You might want to spread out those logfiles over disks/filesystems
- cpu - is it compressing those logs as they are rotated? You might want to use a filesystem with internal compression, and even hardware acceleration.
- directory cache? see the answer by Chris Card - spread your files over directories, and, just as important, make those files differer in the early characters of the filename, or you will not see the benefits of the directory cache applied to them, which slows down lookups significantly.
- if your server is meeting its limits - is it just so for log rotation, or is it already missing out on log entries?
- Are you copying those logfiles around, or are you moving them (within the same filesystem)?
If you design a structure for your logfiles, there's nothing that would keep you from parallelizing rotating them.
edit in reply to added requirements
As far as I know, logrotate does not parallelize in itself. Once you have set up a clear structure though, there are a few roads you can explore to parallelize it (or at least the parts bogging you down) manually:
- The compress command itself: You can use a wrapper script to do the compression. The gzip spawned can run in the background while your script returns immediately - thus leaving the compress job running while logrotate merrily proceeds to the next log file. Two caveats:
- I'd rather be safe and switch on
delaycompress to avoid still fiddling with the logfile while the process is being SIGHUP'ed or whatever it takes to start a new log (depending on the process).
- I'd also "nice" the compress job, to avoid having many parallel compress jobs compete for cpu time.
- If you're feeling sophisticated (or end up with numerous parallel gzip jobs throttling down the loghost), you may later transition to writing a set of worker processes taking their files-to-compress from a list generated by the "compress command" script stated in your logrotate config.
- You can have independent instances of logrotate running with a bit of planning:
- each instance needs its own config file.
- each instance needs its own set of directories to watch for logfiles. You can (and probably should) have separate crontab entries for them.
- each instance needs its own