I'm running a daemon that routinely stats and opens files. It also copies the data from these files over the network to various cloud providers. The daemon runs as a single process. After the process starts I increase the open file limit from the system default of 1024 to 32768 with
prlimit --pid <the process id> --nofile=32768:32768. I've verified that the soft and hard file limits do actually get updated.
Note: When I mention currently open files, I'm referring to the value as returned by
lsof continually running in another window (
while [ true ]; do sudo lsof -p <the process id> | wc -l; done;). It's not just some guess.
The server runs fine for a while, and even under high load it has <3500 files open. However, sometimes under moderate load when only a few hundred files are open (<500) the process starts receiving "too many open files" errors when trying to create sockets, open files, stat files, etc.
Is there some other variable/limit that I'm not accounting for that would cause 'too many open files' even though the soft limit is at 32768 and only a few hundred are actually shown to be open?
- Red Hat Enterprise Linux Server 7.6
- Kernel 3.10.0-957.el7.x86_64 (I know it's ancient. It's not in my control)
Just to be completely clear, by the kernel's own records (via
lsof), the process is not using too many open files. When these errors start occurring the kernel reports only a few hundred open file descriptors (with a process limit of 32768).