2

I'm running a daemon that routinely stats and opens files. It also copies the data from these files over the network to various cloud providers. The daemon runs as a single process. After the process starts I increase the open file limit from the system default of 1024 to 32768 with prlimit --pid <the process id> --nofile=32768:32768. I've verified that the soft and hard file limits do actually get updated.

Note: When I mention currently open files, I'm referring to the value as returned by lsof continually running in another window (while [ true ]; do sudo lsof -p <the process id> | wc -l; done;). It's not just some guess.

The server runs fine for a while, and even under high load it has <3500 files open. However, sometimes under moderate load when only a few hundred files are open (<500) the process starts receiving "too many open files" errors when trying to create sockets, open files, stat files, etc.

Is there some other variable/limit that I'm not accounting for that would cause 'too many open files' even though the soft limit is at 32768 and only a few hundred are actually shown to be open?

Relevant info:

  • Red Hat Enterprise Linux Server 7.6
  • Kernel 3.10.0-957.el7.x86_64 (I know it's ancient. It's not in my control)

Just to be completely clear, by the kernel's own records (via lsof), the process is not using too many open files. When these errors start occurring the kernel reports only a few hundred open file descriptors (with a process limit of 32768).

2
  • Are you using systemd to handle your daemon? Also does your process fork after starting, specifically in regards to when it copies that aforementioned files?
    – ReedGhost
    Jul 1, 2021 at 20:31
  • I'm using systemd, but manually setting the open files limit with prlimit after launch. The daemon never forks. Jul 2, 2021 at 0:34

2 Answers 2

1

I don't know why its doing this, but the first thing I would have down would be to set the ulimit before the daemon is started.

0

Rehashing symcbean's answer

systemd handles limits on a per-process basis, which subsequently means it ignores other limits, so its actually necessary to configure the limit within that service's unit file.

The following is an example:

[Unit]
Description=example systemd service unit file.

[Service]
ExecStart=/bin/bash /usr/sbin/example.sh
LimitNOFILE=32768

[Install]
WantedBy=multi-user.target

Additional Documentaton

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.