Troubleshooting Logrotate

Question

The Problem

I have several identical servers all running identical logrotate configurations rotating a custom log file. On at least one of the servers my logs are not getting rotated correctly.

I have been troubleshooting this problem off and on for weeks now to no avail. Here is some more info and the steps that I have taken so far. Any help would be appreciated.

I am basically running out of ways to troubleshoot this problem and would love some ideas to try to see what is going wrong here.

What I've Tried

(Configuration files are below)

1. Check configuration and permissions on the main logrotate file. Everything looks good to me there.
2. Set permissions on the target directory, derp, to what logrotate needs (the directory can not be world writable afaik).

3. Force logrotate to run with the following:

   logrotate --force /etc/logrotate.d/derp
   

   This command rotates the logs correctly but the next day they will not run.

4. Manually change the date to sometime in the past in /var/lib/logrotate/status and then see if the log gets correctly run the next time logrotate kicks off or with:

   logrotate -vv /etc.logrotate.d/derp
   

   This will rotate the log once but the next time the log is up for rotation it does not get rotated. Here is the output of a successful manual rotate:

   rotating pattern: /derp/*.log  after 1 days (14 rotations)
   empty log files are not rotated, old logs are removed
   considering log /derp/access.log
   log needs rotating
   considering log /derp/php_errors.log
   log does not need rotating
   rotating log /derp/access.log, log->rotateCount is 14
   dateext suffix '-20150814'
   ...
   files getting renamed and moved
   ...
   running prerotate script
   renaming /derp/access.log to /derp/access.log.1
   creating new /derp/access.log mode = 0774 uid = 0 gid = 4
   running postrotate script
   

   Life is good, no? The only problem is that the next day, when it needs rotating (i.e., it always has data and is never empty) nothing happens.

5. I have consulted many SEU questions among which was this question which was helpful in the past dealing with other logrotate issues.

Config

The system is running Debian 7 up to date. The process generating the log files is run as root. The process generating the log files is a php application and spawns multiple children.

Here are the relevant configurations, where derp is the location of my custom log files:

/etc/logrotate.d/derp

Permissions on the config file:

-rw-r--r-- 1 root root 265 Jul 9 2014 derp

/derp/*.log {
                daily
                missingok
                rotate 14
                compress
                delaycompress
                notifempty
                create 774 root adm
                sharedscripts
                postrotate
                                nohup /derp/daemonrestart.sh > /dev/null 2>&1
                endscript
                prerotate
                                killall -9 php > /dev/null 2>&1
                endscript
}

The prerotate and the postrotate scripts basically start the process (a socket server) or stops it and work fine on all of other servers. I am pretty sure that there are no problems there.

/var/lib/logrotate/status

(redacted to show relevant info)

logrotate state -- version 2
"/var/log/kern.log" 2015-7-20
...
"/derp/php_errors.log" 2015-8-11
...
"/derp/access.log" 2015-8-14
...

/derp/

Permissions on the target directory:

drwxr-xr-x 2 root root 4096 Jun 18 10:05 derp

And the relevant log located in \derp\:

-rwxrwxr-- 1 root adm 3755558 Aug 14 07:57 access.log

Answer 1

You wrote:

Force logrotate to run with the following: logrotate --force /etc/logrotate.d/derp. This command rotates the logs correctly but the next day they will not run.

The problem here is that the daily attribute actually means "24 hours". So unless the cron entry for logrotate (probably in /etc/cron.d/daily) is more than 24 hours away from the time you manually rotate the log files, the log files will not be rotated until the day after that.

On my Debian system the entries in /etc/cron.daily are run daily at 06:25, so it's probable that any manual log rotation done during the normal working day will prevent a log rotation occurring until two days later.

Answer 2

Ok I gave up trying to fix this. Cron to the rescue.

sudo crontab -e

Then add the following line to the super user's crontab

# Logrotate they said...
3 6 * * *               /usr/sbin/logrotate -f /etc/logrotate.d/derp &> /dev/null

The -f flag forces logrotate to rotate the logs. The file located at /etc/logrotate.d/derp is the corresponding logrotate configuration for my application.

Its ugly, but it was the only way I could fix this bug.