I was wondering if anything gets logged when any ulimits are hit (open files, etc). If so, where is this logged, and what should I be looking for? I'm using CentOS 6.
You can use audit to log the failures of the corresponding system calls (syscalls), although not every excess manifests this way. For example as
Henk Langeveld pointed out, exceeding
RLIMIT_RTTIME causes the kernel to send a signal.
Let's take for example the
Specifies a value one greater than the maximum file descriptor number that can be opened by this process. Attempts (
dup(2), etc.) to exceed this limit yield the error
EMFILE. (Historically, this limit was named
So you'll have to monitor for example the
open syscall. Its man page says:
creat()return the new file descriptor, or -1 if an error occurred (in which case,
errnois set appropriately).
creat()can fail with the following errors:
EMFILE- The process already has the maximum number of files open.
This means you have to audit the
open syscalls that fail with
EMFILE. The man page suggests that
open returns -1 and sets
EMFILE, but what happens actually is that the
open syscall returns
-EMFILE and glibc converts it into -1 and sets
Now that we have straighten that out, let's add an audit rule:
[root@h ~]# auditctl -a always,exit -F arch=`uname -m` -S open \ -F uid=ciupicri -F exit=-EMFILE -k "UL-SE"
Let's test the limits:
[ciupicri@h ~]$ ulimit -n 10 [ciupicri@h ~]$ python -c 'from __future__ import print_function; f = [(print(i), open("/etc/passwd")) for i in range(10)]' 0 1 2 3 4 5 6 7 Traceback (most recent call last): File "<string>", line 1, in <module> IOError: [Errno 24] Too many open files: '/etc/passwd'
And check the logs:
[root@h ~]# ausearch --start recent -k UL-SE ... time->Wed Jun 25 21:27:37 2014 type=PATH msg=audit(1403720857.418:63): item=0 name="/etc/passwd" nametype=UNK NOWN type=CWD msg=audit(1403720857.418:63): cwd="/home/ciupicri" type=SYSCALL msg=audit(1403720857.418:63): arch=40000003 syscall=5 success=no exit=-24 a0=8ed72e0 a1=8000 a2=1b6 a3=8f24d11 items=1 ppid=1110 pid=1139 auid= 5000 uid=5000 gid=5000 euid=5000 suid=5000 fsuid=5000 egid=5000 sgid=5000 fsgi d=5000 tty=pts3 ses=2 comm="python" exe="/usr/bin/python" subj=unconfined_u:un confined_r:unconfined_t:s0-s0:c0.c1023 key="UL-SE" ...
The "Security Guide" for Red Hat Enterprise Linux 6 has a "System Auditing" chapter where you can read more on the subject.
* Thanks go to fche for pointing this out.
You need to inspect kernel source for this.
The CPU Timers in the linux kernel (posix-cpu-timers.c) will just send a SIGKILL to the offending process when the hard limit is reached. The soft limit triggers an SIGXCPU once a second, and a Watchdog message.
You can search for the other limits and/or signals for more detail.