0

The Problem

I have several identical servers all running identical logrotate configurations rotating a custom log file. On at least one of the servers my logs are not getting rotated correctly.

I have been troubleshooting this problem off and on for weeks now to no avail. Here is some more info and the steps that I have taken so far. Any help would be appreciated.

I am basically running out of ways to troubleshoot this problem and would love some ideas to try to see what is going wrong here.

What I've Tried

(Configuration files are below)

  1. Check configuration and permissions on the main logrotate file. Everything looks good to me there.
  2. Set permissions on the target directory, derp, to what logrotate needs (the directory can not be world writable afaik).
  3. Force logrotate to run with the following:

    logrotate --force /etc/logrotate.d/derp
    

    This command rotates the logs correctly but the next day they will not run.

  4. Manually change the date to sometime in the past in /var/lib/logrotate/status and then see if the log gets correctly run the next time logrotate kicks off or with:

    logrotate -vv /etc.logrotate.d/derp
    

    This will rotate the log once but the next time the log is up for rotation it does not get rotated. Here is the output of a successful manual rotate:

    rotating pattern: /derp/*.log  after 1 days (14 rotations)
    empty log files are not rotated, old logs are removed
    considering log /derp/access.log
    log needs rotating
    considering log /derp/php_errors.log
    log does not need rotating
    rotating log /derp/access.log, log->rotateCount is 14
    dateext suffix '-20150814'
    ...
    files getting renamed and moved
    ...
    running prerotate script
    renaming /derp/access.log to /derp/access.log.1
    creating new /derp/access.log mode = 0774 uid = 0 gid = 4
    running postrotate script
    

    Life is good, no? The only problem is that the next day, when it needs rotating (i.e., it always has data and is never empty) nothing happens.

  5. I have consulted many SEU questions among which was this question which was helpful in the past dealing with other logrotate issues.

Config

The system is running Debian 7 up to date. The process generating the log files is run as root. The process generating the log files is a php application and spawns multiple children.

Here are the relevant configurations, where derp is the location of my custom log files:

/etc/logrotate.d/derp

Permissions on the config file:

-rw-r--r-- 1 root root 265 Jul 9 2014 derp

/derp/*.log {
                daily
                missingok
                rotate 14
                compress
                delaycompress
                notifempty
                create 774 root adm
                sharedscripts
                postrotate
                                nohup /derp/daemonrestart.sh > /dev/null 2>&1
                endscript
                prerotate
                                killall -9 php > /dev/null 2>&1
                endscript
}

The prerotate and the postrotate scripts basically start the process (a socket server) or stops it and work fine on all of other servers. I am pretty sure that there are no problems there.

/var/lib/logrotate/status

(redacted to show relevant info)

logrotate state -- version 2
"/var/log/kern.log" 2015-7-20
...
"/derp/php_errors.log" 2015-8-11
...
"/derp/access.log" 2015-8-14
...

/derp/

Permissions on the target directory:

drwxr-xr-x 2 root root 4096 Jun 18 10:05 derp

And the relevant log located in \derp\:

-rwxrwxr-- 1 root adm 3755558 Aug 14 07:57 access.log

  • logrotate runs via a cron job (see dpkg -L logrotate for the file created in /etc/cron.daily). Is cron running correctly? If you have systemd, can you run systemctl status cron.service and verify that it's running? – saiarcot895 Aug 14 '15 at 14:06
  • I do see a logrotate file inside of /etc/cron.daily/ that looks valid. Additionally when the logs fail to get rotated as expected the date does get updated in /var/lib/logrotate/status. – datUser Aug 14 '15 at 14:13
1

You wrote:

Force logrotate to run with the following: logrotate --force /etc/logrotate.d/derp. This command rotates the logs correctly but the next day they will not run.

The problem here is that the daily attribute actually means "24 hours". So unless the cron entry for logrotate (probably in /etc/cron.d/daily) is more than 24 hours away from the time you manually rotate the log files, the log files will not be rotated until the day after that.

On my Debian system the entries in /etc/cron.daily are run daily at 06:25, so it's probable that any manual log rotation done during the normal working day will prevent a log rotation occurring until two days later.

  • So the logs never rotate correctly whether or not I do the logrotate --force or not. I have also manually set the values in /var/lib/logrotate/status to indicate that the log needs rotating. Still nothing. This issue began to show up about 6 months ago on new systems created from a FAI (Fully Automated Install) install of linux. The last 10 systems out of over 40 started exhibiting this behavior. – datUser Dec 8 '15 at 18:14
  • @datUser incorrect date/time on the suspect systems? – roaima Dec 9 '15 at 16:35
0

Ok I gave up trying to fix this. Cron to the rescue.

sudo crontab -e

Then add the following line to the super user's crontab

# Logrotate they said...
3 6 * * *               /usr/sbin/logrotate -f /etc/logrotate.d/derp &> /dev/null

The -f flag forces logrotate to rotate the logs. The file located at /etc/logrotate.d/derp is the corresponding logrotate configuration for my application.

Its ugly, but it was the only way I could fix this bug.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.