Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I was wondering if anything gets logged when any ulimits are hit (open files, etc). If so, where is this logged, and what should I be looking for? I'm using CentOS 6.

share|improve this question
1  
For what operating system, exactly? If something like this exists, it's probably OS specific. –  godlygeek Jun 24 at 20:15
    
Those ulimits make some syscalls fail if their limits are reached, nothing more. Unless there's some auditing framework for this (audit perhaps?), you would probably have to monitor the applications with something like strace, SystemTap, etc. –  Cristian Ciupitu Jun 24 at 21:13
    
The limits are always exist, even if not set. An application is able and can handle them. As @CristianCiupitu stated, you can monitor every failed system call with auditd. –  user55518 Jun 24 at 22:40

2 Answers 2

You can use audit to log the failures of the corresponding system calls (syscalls), although not every excess manifests this way. For example as Henk Langeveld pointed out, exceeding RLIMIT_RTTIME causes the kernel to send a signal.

Let's take for example the RLIMIT_NOFILE limit:

Specifies a value one greater than the maximum file descriptor number that can be opened by this process. Attempts (open(2), pipe(2), dup(2), etc.) to exceed this limit yield the error EMFILE. (Historically, this limit was named RLIMIT_OFILE on BSD.)

So you'll have to monitor for example the open syscall. Its man page says:

RETURN VALUE

open(), openat(), and creat() return the new file descriptor, or -1 if an error occurred (in which case, errno is set appropriately).

ERRORS

open(), openat(), and creat() can fail with the following errors:

EMFILE - The process already has the maximum number of files open.

This means you have to audit the open syscalls that fail with EMFILE. The man page suggests that open returns -1 and sets errno to EMFILE, but what happens actually is that the open syscall returns -EMFILE and glibc converts it into -1 and sets errno to EMFILE *.

Now that we have straighten that out, let's add an audit rule:

[root@h ~]# auditctl -a always,exit -F arch=`uname -m` -S open \
                                    -F uid=ciupicri -F exit=-EMFILE -k "UL-SE"

Let's test the limits:

[ciupicri@h ~]$ ulimit -n 10
[ciupicri@h ~]$ python -c 'from __future__ import print_function; f = [(print(i), open("/etc/passwd")) for i in range(10)]'
0
1
2
3
4
5
6
7
Traceback (most recent call last):
  File "<string>", line 1, in <module>
IOError: [Errno 24] Too many open files: '/etc/passwd'

And check the logs:

[root@h ~]# ausearch --start recent -k UL-SE
...
time->Wed Jun 25 21:27:37 2014
type=PATH msg=audit(1403720857.418:63): item=0 name="/etc/passwd" nametype=UNK
NOWN
type=CWD msg=audit(1403720857.418:63):  cwd="/home/ciupicri"
type=SYSCALL msg=audit(1403720857.418:63): arch=40000003 syscall=5 success=no 
exit=-24 a0=8ed72e0 a1=8000 a2=1b6 a3=8f24d11 items=1 ppid=1110 pid=1139 auid=
5000 uid=5000 gid=5000 euid=5000 suid=5000 fsuid=5000 egid=5000 sgid=5000 fsgi
d=5000 tty=pts3 ses=2 comm="python" exe="/usr/bin/python" subj=unconfined_u:un
confined_r:unconfined_t:s0-s0:c0.c1023 key="UL-SE"
...

The "Security Guide" for Red Hat Enterprise Linux 6 has a "System Auditing" chapter where you can read more on the subject.

* Thanks go to fche for pointing this out.

share|improve this answer

You need to inspect kernel source for this.

Example:

The CPU Timers in the linux kernel (posix-cpu-timers.c) will just send a SIGKILL to the offending process when the hard limit is reached. The soft limit triggers an SIGXCPU once a second, and a Watchdog message.

You can search for the other limits and/or signals for more detail.

share|improve this answer
    
With thanks to the Linux Cross Reference project –  Henk Langeveld Jun 24 at 22:42

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.