2

Background: I have a CentOS 6 LAMP server. Recently the server has started to become unresponsive every few days. Originally, mysqld would throw a nagios alert and I would be unable to even ssh into the server, a hard reset was necessary. Mysqltuner lead me to increase the buffer pool, which seemed to help. Now the symptom has changed to nagios throwing an apache http down alert. I was able to ssh into the server this time but apache failed to restart and a reboot was necessary.

After looking at /var/log/messages and /var/log/audit/audit.log I see that there are hundreds of AVC errors. audit.log is several MB daily, while my other servers are just kb in size. Could this be a clue to the underlying problem?

A typical /var/log/messages entry is this:

Mar 31 16:50:39 web1 setroubleshoot: SELinux is preventing /bin/ps from getattr access on the directory /proc/<pid>. For complete SELinux messages. run sealert -l be51d126-d70e-491f-9ec8-f897677d9989

Running it through sealert yields the following:

SELinux is preventing /bin/ps from getattr access on the directory /proc/<pid>.

*****  Plugin catchall (100. confidence) suggests  ***************************

If you believe that ps should be allowed getattr access on the <pid> directory by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# grep ps /var/log/audit/audit.log | audit2allow -M mypol
# semodule -i mypol.pp

Here's a typical entry in audit.log:

type=SYSCALL msg=audit(1427837702.229:721164): arch=c000003e syscall=4 success=no exit=-13 a0=8164d0 a1=3eaee11cc0 a2=
3eaee11cc0 a3=8164d6 items=0 ppid=2792 pid=2800 auid=4294967295 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48
 fsgid=48 tty=(none) ses=4294967295 comm="ps" exe="/bin/ps" subj=system_u:system_r:httpd_t:s0 key=(null)
type=AVC msg=audit(1427837702.219:721127): avc:  denied  { getattr } for  pid=2800 comm="ps" path="/proc/875" dev=proc
 ino=9349054 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:kernel_t:s0 tclass=dir

UPDATE Ok, months later it's happened again. I'm no closer to figuring out why my LAMP server is freezing up from time to time (I suspect MySQL since that is the first service to throw a nagios alert), but I know why the SE Linux alerts (from my original question) are happening: one of the sites hosted is a Magento online store, and the cron.php script that fires every five minutes is causing the SE Linux errors, every time.

So my updated question is: is this something to worry about, other than the massive amounts of entries in my messages and and audit logs?

  • 2
    Did you run audit2why already on the audit lines? It describes the reason for SELinux denying the actions. In that case httpd wants to run ps but is not allowed to (I guess nagios is the initiator of that). Anyway, I do not think that is the reason for your troubles. What about resources (mem/io)? – Manuel Faux Apr 2 '15 at 9:32
  • Thanks for the comment. Seems strange that there are so many AVC events logged. I'll try to capture more information about mem/io the next time it happens, too, thanks. – Rocky Apr 2 '15 at 19:03
  • 1
    It's denying the access that's all we can know. What it means that php isn't able to execute ps is dependent on what running ps was supposed to accomplish. It's impossible for third parties to really know the answer to this question. Following the error's instructions will correct the error, otherwise you can configure it as dontaudit to keep it out of your logs. – Bratchley May 24 '15 at 20:25
2

I was finally was able to narrow down and solve the problem. It was a combination of two issues:

  1. A Magento site on the server was disabled in the vhosts file. But the Magento cron job was still running, failing, and causing all the AVC errors. Removing the orphaned cron job stopped the AVC errors.

  2. As Manuel Faux suggested in the comments, however, the SELinux errors were unrelated to the random crashes of the server. But with the AVC entries no longer cluttering up my log files, I was able to locate the following in the mysql logs just before the server freezes:

    InnoDB: Warning: a long semaphore wait: --Thread 140485795231488 has waited at btr0sea.c line 1706 for 241.00 seconds the semaphore: X-lock on RW-latch at 0x5583b18 created in file btr0sea.c line 178

Those logs about the semaphore wait led me to this related question. So the final solution was to set innodb_adaptive_hash_index = 0 in the mysql config.

As a further step I also created a weekly mysqlcheck to optimize all databases. It's been several weeks now and no spontaneous crashes and no more crazy error logs for mysql or SELinux.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.