The Problem
I have several identical servers all running identical logrotate
configurations rotating a custom log file. On at least one of the servers my logs are not getting rotated correctly.
I have been troubleshooting this problem off and on for weeks now to no avail. Here is some more info and the steps that I have taken so far. Any help would be appreciated.
I am basically running out of ways to troubleshoot this problem and would love some ideas to try to see what is going wrong here.
What I've Tried
(Configuration files are below)
- Check configuration and permissions on the main
logrotate
file. Everything looks good to me there. - Set permissions on the target directory,
derp
, to what logrotate needs (the directory can not be world writable afaik). Force
logrotate
to run with the following:logrotate --force /etc/logrotate.d/derp
This command rotates the logs correctly but the next day they will not run.
Manually change the date to sometime in the past in
/var/lib/logrotate/status
and then see if the log gets correctly run the next timelogrotate
kicks off or with:logrotate -vv /etc.logrotate.d/derp
This will rotate the log once but the next time the log is up for rotation it does not get rotated. Here is the output of a successful manual rotate:
rotating pattern: /derp/*.log after 1 days (14 rotations) empty log files are not rotated, old logs are removed considering log /derp/access.log log needs rotating considering log /derp/php_errors.log log does not need rotating rotating log /derp/access.log, log->rotateCount is 14 dateext suffix '-20150814' ... files getting renamed and moved ... running prerotate script renaming /derp/access.log to /derp/access.log.1 creating new /derp/access.log mode = 0774 uid = 0 gid = 4 running postrotate script
Life is good, no? The only problem is that the next day, when it needs rotating (i.e., it always has data and is never empty) nothing happens.
I have consulted many SEU questions among which was this question which was helpful in the past dealing with other logrotate issues.
Config
The system is running Debian 7 up to date. The process generating the log files is run as root. The process generating the log files is a php
application and spawns multiple children.
Here are the relevant configurations, where derp
is the location of my custom log files:
/etc/logrotate.d/derp
Permissions on the config file:
-rw-r--r-- 1 root root 265 Jul 9 2014 derp
/derp/*.log {
daily
missingok
rotate 14
compress
delaycompress
notifempty
create 774 root adm
sharedscripts
postrotate
nohup /derp/daemonrestart.sh > /dev/null 2>&1
endscript
prerotate
killall -9 php > /dev/null 2>&1
endscript
}
The prerotate
and the postrotate
scripts basically start the process (a socket server) or stops it and work fine on all of other servers. I am pretty sure that there are no problems there.
/var/lib/logrotate/status
(redacted to show relevant info)
logrotate state -- version 2
"/var/log/kern.log" 2015-7-20
...
"/derp/php_errors.log" 2015-8-11
...
"/derp/access.log" 2015-8-14
...
/derp/
Permissions on the target directory:
drwxr-xr-x 2 root root 4096 Jun 18 10:05 derp
And the relevant log located in \derp\
:
-rwxrwxr-- 1 root adm 3755558 Aug 14 07:57 access.log
logrotate
runs via a cron job (seedpkg -L logrotate
for the file created in/etc/cron.daily
). Is cron running correctly? If you have systemd, can you runsystemctl status cron.service
and verify that it's running?logrotate
file inside of/etc/cron.daily/
that looks valid. Additionally when the logs fail to get rotated as expected the date does get updated in/var/lib/logrotate/status
.