3

I am running LXC on Archlinux.

I have a combination of Arch, Fedora, Centos, Debian, and Ubuntu LXC containers, each based on systemd.

If I try to open more than 19 LXC containers, the 20th container will start, but without systemd running inside. (ps aux shows just bash, init, and ps)

I thought this might be related to the number of open files, since a centos container will sometimes report "too many open files" when I run poweroff inside of it while I have a large number of LXC containers running. But I increased the file limits as described in this link, rebooted and verified the changes, but my problem persists.

What could be causing this?

7
  • 1
    You need to explain in the question how you know that that init is not systemd, and what program it instead is.
    – JdeBP
    Aug 28, 2018 at 7:00
  • @JdeBP I guess I didn't know that init could be systemd. But when I run ps aux on the same container while everything works, I see a large number of other processes like systemd-journald, systemd-networkd, and importantly for my purposes, /usr/bin/sshd -D which is started by systemd. When things are not working, nothing related to systemd works, such as systemctl status sshd or poweroff. I just get this error: Failed to connect to bus: No such file or directory.
    – rexroni
    Aug 28, 2018 at 13:39
  • Which Linux variant is your 20th container running, and how much system memory is available before after it starts? Aug 28, 2018 at 19:53
  • 1
    @MarkStosberg I confirmed the same behavior where the 20th container was Fedora, Arch, or Ubuntu, and sometimes (but not always) Debian. As far as memory goes, I am pretty sure I was running at around 15GB of memory available (out of 32GB, but no swap enabled). I can't check because today the problem seems to not want to reproduce. This has been bugging me for a couple weeks and when I got it to reproduce through multiple reboots I posted the question... but the issue seems to be not reproducing at the moment *deep, dejected sigh*
    – rexroni
    Aug 29, 2018 at 2:57
  • 2
    Does this answer your question? "Error: Too many open files" while starting service in environment with several LXCs
    – A.B
    May 26, 2023 at 10:27

1 Answer 1

1

If your problem happens with non-privileged (ie: using user namespaces) containers, rather than privileged (ie: root) containers, I believe this would be caused by a restricted limit on the use of the inotify interface for the same user running containers. Apparently systemd relies on inotify. The Debian buster LXC package includes a setting loosening the sysctl fs.inotify.max_user_instances in /etc/sysctl.d/30-lxc-inotify.conf:

# Defines the maximum number of inotify listeners.
# By default, this value is 128, which is quickly exhausted when using
# systemd-based LXC containers (15 containers are enough).
# When the limit is reached, systemd becomes mostly unusable, throwing
# "Too many open files" all around (both on the host and in containers).
# See https://kdecherf.com/blog/2015/09/12/systemd-and-the-fd-exhaustion/
# Increase the user inotify instance limit to allow for about
# 100 containers to run before the limit is hit again
fs.inotify.max_user_instances = 1024

The relevant documentation for this setting tells:

/proc/sys/fs/inotify/max_user_instances
This specifies an upper limit on the number of inotify instances that can be created per real user ID.

If your LXC installation doesn't include such a setting, you could check if simply running on the host:

sysctl -w fs.inotify.max_user_instances=1024

will fix the problem.

2
  • Neat! I am not currently in a position to reproduce the bug and check if this works, but I will come back and mark the answer as correct if it does.
    – rexroni
    Jun 3, 2019 at 2:00
  • I just found after answering unix.stackexchange.com/questions/747085/… that I already made an answer here. Anyway here OP never validated this answer and appears MIA while my new answer has been (and has more information about root/non-root). So deleting this answer before voting to have the (old) question set as duplicate of the new question.
    – A.B
    May 26, 2023 at 10:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .