I am trying to find the reason why my long-running app sometimes busts the maximum open file descriptor limit (ulimit -n). I would like to periodically log how many file descriptors the app has open so that I can see when the spike occurred.
I know that lsof includes a bunch of items that are excluded from /proc/$PID/fd... Are those items relevant with regard to the open file descriptor limit? I.e. should I be logging info from lsof or from /proc/$PID/fd?
tl;dr ls -U /proc/PID/fd | wc -l will tell you the number that should be less than ulimit -n.
/proc/PID/fd should contain all the file descriptors opened by a process, including but not limited to strange ones like epoll or inotify handles, "opaque" directory handles opened with O_PATH, handles opened with signalfd() or memfd_create(), sockets returned by accept(), etc.
I'm not a great lsof user, but lsof is getting its info from /proc, too. I don't think there's another way to get the list of the file descriptors a process has opened on Linux other than procfs, or by attaching to a process with ptrace.
Anyways, the current and root directory, mmapped files (including its own binary and dynamic libraries) and controlling terminal of a process are not counted against the limit set with ulimit -n (RLIMIT_NOFILE), and they also don't appear in /proc/PID/fd unless the process is explicitly holding open handles to them.
sysctl fs.file-nr | awk ' { print $3 } '– Rui F Ribeiro Feb 8 at 18:24