Too many open files despite a sufficiently high open files limit

Question

I'm running a daemon that routinely stats and opens files. It also copies the data from these files over the network to various cloud providers. The daemon runs as a single process. After the process starts I increase the open file limit from the system default of 1024 to 32768 with prlimit --pid <the process id> --nofile=32768:32768. I've verified that the soft and hard file limits do actually get updated.

Note: When I mention currently open files, I'm referring to the value as returned by lsof continually running in another window (while [ true ]; do sudo lsof -p <the process id> | wc -l; done;). It's not just some guess.

The server runs fine for a while, and even under high load it has <3500 files open. However, sometimes under moderate load when only a few hundred files are open (<500) the process starts receiving "too many open files" errors when trying to create sockets, open files, stat files, etc.

Is there some other variable/limit that I'm not accounting for that would cause 'too many open files' even though the soft limit is at 32768 and only a few hundred are actually shown to be open?

Relevant info:

• Red Hat Enterprise Linux Server 7.6
• Kernel 3.10.0-957.el7.x86_64 (I know it's ancient. It's not in my control)

Just to be completely clear, by the kernel's own records (via lsof), the process is not using too many open files. When these errors start occurring the kernel reports only a few hundred open file descriptors (with a process limit of 32768).

Answer 1

I don't know why its doing this, but the first thing I would have down would be to set the ulimit before the daemon is started.

Answer 2

Rehashing symcbean's answer

systemd handles limits on a per-process basis, which subsequently means it ignores other limits, so its actually necessary to configure the limit within that service's unit file.

The following is an example:

[Unit]
Description=example systemd service unit file.

[Service]
ExecStart=/bin/bash /usr/sbin/example.sh
LimitNOFILE=32768

[Install]
WantedBy=multi-user.target

Additional Documentaton