I was wondering if anything gets logged when any ulimits are hit (open files, etc). If so, where is this logged, and what should I be looking for? I'm using CentOS 6.

  • 1
    For what operating system, exactly? If something like this exists, it's probably OS specific. – godlygeek Jun 24 '14 at 20:15
  • Those ulimits make some syscalls fail if their limits are reached, nothing more. Unless there's some auditing framework for this (audit perhaps?), you would probably have to monitor the applications with something like strace, SystemTap, etc. – Cristian Ciupitu Jun 24 '14 at 21:13
  • The limits are always exist, even if not set. An application is able and can handle them. As @CristianCiupitu stated, you can monitor every failed system call with auditd. – user55518 Jun 24 '14 at 22:40

You can use audit to log the failures of the corresponding system calls (syscalls), although not every excess manifests this way. For example as Henk Langeveld pointed out, exceeding RLIMIT_RTTIME causes the kernel to send a signal.

Let's take for example the RLIMIT_NOFILE limit:

Specifies a value one greater than the maximum file descriptor number that can be opened by this process. Attempts (open(2), pipe(2), dup(2), etc.) to exceed this limit yield the error EMFILE. (Historically, this limit was named RLIMIT_OFILE on BSD.)

So you'll have to monitor for example the open syscall. Its man page says:


open(), openat(), and creat() return the new file descriptor, or -1 if an error occurred (in which case, errno is set appropriately).


open(), openat(), and creat() can fail with the following errors:

EMFILE - The process already has the maximum number of files open.

This means you have to audit the open syscalls that fail with EMFILE. The man page suggests that open returns -1 and sets errno to EMFILE, but what happens actually is that the open syscall returns -EMFILE and glibc converts it into -1 and sets errno to EMFILE *.

Now that we have straighten that out, let's add an audit rule:

[root@h ~]# auditctl -a always,exit -F arch=`uname -m` -S open \
                                    -F uid=ciupicri -F exit=-EMFILE -k "UL-SE"

Let's test the limits:

[ciupicri@h ~]$ ulimit -n 10
[ciupicri@h ~]$ python -c 'from __future__ import print_function; f = [(print(i), open("/etc/passwd")) for i in range(10)]'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
IOError: [Errno 24] Too many open files: '/etc/passwd'

And check the logs:

[root@h ~]# ausearch --start recent -k UL-SE
time->Wed Jun 25 21:27:37 2014
type=PATH msg=audit(1403720857.418:63): item=0 name="/etc/passwd" nametype=UNK
type=CWD msg=audit(1403720857.418:63):  cwd="/home/ciupicri"
type=SYSCALL msg=audit(1403720857.418:63): arch=40000003 syscall=5 success=no 
exit=-24 a0=8ed72e0 a1=8000 a2=1b6 a3=8f24d11 items=1 ppid=1110 pid=1139 auid=
5000 uid=5000 gid=5000 euid=5000 suid=5000 fsuid=5000 egid=5000 sgid=5000 fsgi
d=5000 tty=pts3 ses=2 comm="python" exe="/usr/bin/python" subj=unconfined_u:un
confined_r:unconfined_t:s0-s0:c0.c1023 key="UL-SE"

The "Security Guide" for Red Hat Enterprise Linux 6 has a "System Auditing" chapter where you can read more on the subject.

* Thanks go to fche for pointing this out.


You need to inspect kernel source for this.


The CPU Timers in the linux kernel (posix-cpu-timers.c) will just send a SIGKILL to the offending process when the hard limit is reached. The soft limit triggers an SIGXCPU once a second, and a Watchdog message.

You can search for the other limits and/or signals for more detail.

  • With thanks to the Linux Cross Reference project – Henk Langeveld Jun 24 '14 at 22:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.