We encountered an issue in production, I suspect it may be caused by hitting the nofile limit, but I cannot find trace or direct proof. We are running on RHEL 7. I used lsof command below to find number of open file descriptors by the user.

/sbin/lsof -u <issue_user> | awk '$4 ~ /[0-9]+[rwu -].*/'

The awk part of this command is to get only file descriptors starting with integers, and ignore mem/rtd/cwd and etc. The above command returned more than 400,000 results, but in /etc/security/limits.conf, the user's nofile limit is 131072.

cat /etc/security/limits.conf
<issue_user>  hard    nofile  131072
<issue_user>  soft    nofile  131072

Running ulimit -aH as the issue user:

$ ulimit -aH
address space limit (Kibytes)  (-M)  unlimited
core file size (blocks)        (-c)  unlimited
cpu time (seconds)             (-t)  unlimited
data size (Kibytes)            (-d)  unlimited
file size (blocks)             (-f)  unlimited
locks                          (-x)  unlimited
locked address space (Kibytes) (-l)  unlimited
message queue size (Kibytes)   (-q)  800
nice                           (-e)  0
nofile                         (-n)  131072
nproc                          (-u)  131072
pipe buffer size (bytes)       (-p)  4096
max memory size (Kibytes)      (-m)  unlimited
rtprio                         (-r)  0
socket buffer size (bytes)     (-b)  4096
sigpend                        (-i)  724561
stack size (Kibytes)           (-s)  32768
swap size (Kibytes)            (-w)  not supported
threads                        (-T)  not supported
process size (Kibytes)         (-v)  unlimited

According to /etc/security/limits.conf, nofile is "max number of open file descriptors". Why this user can have more open file descriptors than defined in limits.conf?

  • 2
    That limit is per user session, not per-user
    – jordanm
    Apr 22, 2020 at 18:45
  • 1
    Also, it's not used in cases where session start doesn't occur via PAM (eg systemd services, which has it's own file limit mechanism)
    – jordanm
    Apr 22, 2020 at 18:47
  • Thank you @jordanm! I don't know the limit is per user session, this is great information! Do you have any document or reference so I can read more about it?
    – Patrick
    Apr 22, 2020 at 18:55
  • Manpages for pam and pam_limits. There are likely more in-depth docs on pam out there as well.
    – jordanm
    Apr 22, 2020 at 19:02


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.