Error stat of log failed: Permission denied during LogRotation

Question

After installing a new CentOS 6.0 server, logrotate was working absolutely fine. Then one day due to a kernel panic, the server had to be hard booted, and ever since log rotation is not rotating the logs.

So I did a separate cron entry to rotate logs manually and forcefully and redirected the output to a log file, and got the following lines for each file:

rotating pattern: /home/mail3/log/popMailProcessing.log  forced from command line (60 rotations)
empty log files are rotated, old logs are removed
considering log /home/mail3/log/popMailProcessing.log
error: stat of /home/mail3/log/popMailProcessing.log failed: Permission denied

However, if I do a logrotation manually from command line, it works flawlessly. The command I use on command line is:

logrotate -v -f /etc/logrotate.d/mail3-logs

My logrotate.conf file is:

# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
dateext

# uncomment this if you want your log files compressed
compress

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp and btmp -- we'll rotate them here
/var/log/wtmp {
    monthly
    create 0664 root utmp
        minsize 1M
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0600 root utmp
    rotate 1
}

# system-specific logs may be also be configured here.

The log rotation file which logrotate uses via cron job is:

dateext
/home/mail3/log/pop.log {
        daily
        rotate 60
        copytruncate
        compress
}
/home/mail3/log/oc4j.log {
        daily
        rotate 60
        copytruncate
        compress
}
/home/mail3/log/incoming.log {
        daily
        rotate 60
        copytruncate
        compress
}
/home/mail3/log/mailpro.log {
        daily
        rotate 60
        copytruncate
        compress
}
/home/mail3/log/imap.log {
        daily
        rotate 60
        copytruncate
        compress
}
/home/mail3/log/outgoing.log {
        daily
        rotate 60
        copytruncate
        compress
}
/home/mail3/log/smtpout.log {
        daily
        rotate 60
        copytruncate
        compress
}
/home/mail3/log/retry.log {
        daily
        rotate 60
        copytruncate
        compress
}
/home/mail3/log/mailinglist.log {
        daily
        rotate 60
        copytruncate
        compress
}
/home/mail3/log/migrate.log {
        daily
        rotate 60
        copytruncate
        compress
}

My crontab entry is:

03 00 * * * root /usr/sbin/logrotate -f -v /etc/logrotate.d/mail3-logs &>> /var/log/logrotate/rotate.log

SELinux is enforcing, and it was enforcing prior to the hard boot too. The directory where the logs are kept have the root as their owner and directory has complete permissions.

Any clue what is causing the permission denied error?

Answer 1

Your original error messages make no sense with what you're showing for your cron that runs your logrotate.

rotating pattern: /home/mail3/log/popMailProcessing.log  forced from command line (60 rotations)
empty log files are rotated, old logs are removed
considering log /home//log/popMailProcessing.log
error: stat of /home/mail3/log/popMailProcessing.log failed: Permission denied

What are these paths doing going to /home/mail3/log/*? Also what's missing from the /home//log/popMailProcessing.log line? Seems like you're only showing some of the actual situation in your question.

Debugging the issue

Put this line in a shell script, logrotate.sh:

#!/bin/bash
/usr/sbin/logrotate -f -v /etc/logrotate.d/mail3-logs &>> /var/log/logrotate/rotate.log

Make it executable and run it like this from the cron:

03 00 * * * root strace -s 2000 -o /tmp/strace.log /path/to/logrotate.bash

In going through the output you should see what is getting tripped up by the permissions problems.

EDIT #1

After conversing with the OP he mentioned that the above debugging technique uncovered that SELinux was enabled. He was perplexed as to why this was the case since he had previously disabled it with the command setenforce 0.

Disabling SELinux in this fashion will only remain in this state until the next reboot. The default mode for SELinux is dictated by this file on Fedora/CentOS:

$ cat /etc/sysconfig/selinux
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#   enforcing - SELinux security policy is enforced.
#   permissive - SELinux prints warnings instead of enforcing.
#   disabled - SELinux is fully disabled.
SELINUX=disabled
# SELINUXTYPE= type of policy in use. Possible values are:
#   targeted - Only targeted network daemons are protected.
#   strict - Full SELinux protection.
SELINUXTYPE=targeted

To permanently disable SELinux you'll want to change the line SELINUX=.. to one of the 3 states, enforcing, permissive, disabled.

I would encourage you however to take the time to understand why SELinux is disallowing the access to the directory these log files are within, and add the appropriate context's so that SELinux allows this access. SELinux is an important part of the layered security model that is facilitated on Linux distros that make use of it, and blindly disabling it is taking one of the critical layers away.

References

• 45.2.7. Enable or Disable SELinux - CentOS 5 user's guide
• 5.4. Enabling and Disabling SELinux - Red Hat 6 user's guide

Answer 2

I think disabling SELinux is not the best option. The better solution, in my opinion, is to create and apply policy. Here is an example of how to do it for other policy http://www.greenvalleyconsulting.org/2015/01/28/installing-coldfusion-11-on-centos-6-6-with-selinux-enforcing/. The same concept will apply to logrotate_t policy instead of httpd_t as outlined in link.

See steps to install policycoreutils-python in the link. then run

grep logrotate /var/log/audit/audit.log | audit2why

audit2allow -a

look for logrotate_t, it more likely will look like this

#============= logrotate_t ============== 
allow logrotate_t file_t:file getattr;

then run

audit2allow -a -M logrotate_t

semodule -i logrotate_t.pp

chcon -R -t logrotate_t /[your log file location]/*.log

Answer 3

In a pinch, you can work around this with:

semanage permissive -a logrotate_t

That's offered in the logrotate_selinux man page.

You'll still be able to see the messages (that you should eventually address) with audit2allow.

Answer 4

We had this error because su group user was used outside of the {…} blocks. It looks like it created conflicts for other rules in /etc/logrotate.d/.

We fixed it by using su root root and create 664 www-data www-data in the rule for the files owned by the www-data user.

So that root rotate the files but www-data own the base file.