9

Environment:

I am using a CentOS-7 as a hypervisor for running several LXCs under libvirt. Each container runs a minimal installation of CentOS-7 with cut down FreePBX (Asterisk, Apache, MySQL + bits).

Symptoms:

There are 16 containers running without any problems. When I start one more it does start, but after the 17th container starts I can not do systemctl start/restart/stop <anything> in ANY of the containers:

[root@test-lxc ~]# systemctl restart dnsmasq
Error: Too many open files

Diagnostics:

The following diagnostics and counts are done while the 17th LXC is running and systemctl restart blabla is failing:

I can ssh into any LXC and run most basic commands, e.g. ls, etc. I suspect the limit somehow affects only the systemd.

I'm trying to understand where/why I hit the limit.

[root@lxc-hypervisor]# sysctl fs.file-nr
fs.file-nr = 29616      0       12988463

That was not tweaked, this is just what happened to be from the default install. Same as above maximum (last) value = 12988463 is reported by the hypervisor and also inside each LXC. Very similar 1st value just under 30000 is also reported in each LXC.

When I try to count file descriptors across all process inside each LXC I get in the order 400 ~ 500 in each LXC.

for pid in $( ls /proc/ | grep -E -e "^[0-9][0-9]*\$" ); do
    ls -l /proc/${pid}/fd/ 2> /dev/null | wc -l
done

The sum total around 9000 (9k) without the hypervisor itself. When I run that on the hypervisor I usually get suspiciously close values just over 10000 e.g. 10005.

Questions:

Q1. Where is the limit set or inherited from?

Q2. Why does the limit affect systemctl start/stop/restart blah commands, but I can still ssh into LXCs, run commands such as bash scripts with loops that fork a lot, albeit as root.

Q3. How to tweak limits to allow running more LXCs. To the best of my understanding RAM and other resources are not the limit.

I did read many articles and answers on the subject of file descriptor limits, but I fail to see where my system hits the limits.

Any other relevant information is also welcome.

2
  • Can you share with us the ulimit -Hn output?
    – K-attila-
    May 26, 2023 at 8:50
  • @K-att- on the host/hypervisor from a ssh root shell the ulimit -Hh outputs 4096 and on the LXC it outputs 8192. A.B's answer solved my problem, but I am happy to explore other queries and answer more comments. Thank you.
    – AnyDev
    May 26, 2023 at 10:27

1 Answer 1

11

I believe you are not hitting a global limit, but an inotify limit. This would be seen on containers running systemd because systemd uses the inotify facility for its bookkeeping, but the host would also be affected. Containers not using systemd (nor inotify) would probably be unaffected.

/proc/sys/fs/inotify/max_user_instances:

This specifies an upper limit on the number of inotify instances that can be created per real user ID.

If only non-rootless (ie: root in the container is the real root) containers are in use, then root user becomes the bottleneck. Having multiple containers using the same rootless user mapping would also create such bottleneck for this container's root user (but not affect the host). The default is 128, far too little for containers use.

CentOS7 (or Rocky9) doesn't include any default setup for this with LXC. Debian-based distributions include this file on the host:

/etc/sysctl.d/30-lxc-inotify.conf:

# Defines the maximum number of inotify listeners.
# By default, this value is 128, which is quickly exhausted when using
# systemd-based LXC containers (15 containers are enough).
# When the limit is reached, systemd becomes mostly unusable, throwing
# "Too many open files" all around (both on the host and in containers).
# See https://kdecherf.com/blog/2015/09/12/systemd-and-the-fd-exhaustion/
# Increase the user inotify instance limit to allow for about
# 100 containers to run before the limit is hit again
fs.inotify.max_user_instances = 1024

So you should do the same by creating this file on the host. For immediate effect (on the host):

sysctl -w fs.inotify.max_user_instances=1024
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .