I'm managing a server which works as a remote log repository for huge number of files. Currently a single
logrotate instance cannot keep up with this number of logs. What's the best way to accelerate log rotation, possibly running many instances in parallel?
Below is the addendum for the clarification:
- Disk I/O is not the problem; also, I'm not losing any logs or entries.
- System is a beast when compared to this job (it's a quad core Xeon with sufficient power) the bottleneck is at compression. One core tops up w/ 1.00 load and cannot finish its compression queue.
- Logfiles are not copied. Only a daily backup is taken to another disk, but again is not a bottleneck.
- I can group the logfiles by patterns. Files are not huge, only they are a lot. I just don't know how to parallelize the logrotate altough I have ideas but unable to find any resources for it. Actually, the core of the question is does logrotate spawn a child process for every stub in the config file or configuration file?
- I'm a computer engineer and an experienced Linux user and developer. I also manage a cluster but somehow I'm new at "old school system management" issues. So you can go as technical / hardcore as you can get. I'll do my best to understand it.