We encountered an issue in production, I suspect it may be caused by hitting the nofile limit, but I cannot find trace or direct proof. We are running on RHEL 7. I used lsof
command below to find number of open file descriptors by the user.
/sbin/lsof -u <issue_user> | awk '$4 ~ /[0-9]+[rwu -].*/'
The awk
part of this command is to get only file descriptors starting with integers, and ignore mem/rtd/cwd and etc. The above command returned more than 400,000 results, but in /etc/security/limits.conf
, the user's nofile limit is 131072.
cat /etc/security/limits.conf
...
<issue_user> hard nofile 131072
<issue_user> soft nofile 131072
...
Running ulimit -aH
as the issue user:
$ ulimit -aH
address space limit (Kibytes) (-M) unlimited
core file size (blocks) (-c) unlimited
cpu time (seconds) (-t) unlimited
data size (Kibytes) (-d) unlimited
file size (blocks) (-f) unlimited
locks (-x) unlimited
locked address space (Kibytes) (-l) unlimited
message queue size (Kibytes) (-q) 800
nice (-e) 0
nofile (-n) 131072
nproc (-u) 131072
pipe buffer size (bytes) (-p) 4096
max memory size (Kibytes) (-m) unlimited
rtprio (-r) 0
socket buffer size (bytes) (-b) 4096
sigpend (-i) 724561
stack size (Kibytes) (-s) 32768
swap size (Kibytes) (-w) not supported
threads (-T) not supported
process size (Kibytes) (-v) unlimited
According to /etc/security/limits.conf
, nofile is "max number of open file descriptors". Why this user can have more open file descriptors than defined in limits.conf
?
pam
andpam_limits
. There are likely more in-depth docs on pam out there as well.