I am running LXC on Archlinux.

I have a combination of Arch, Fedora, Centos, Debian, and Ubuntu LXC containers, each based on systemd.

If I try to open more than 19 LXC containers, the 20th container will start, but without systemd running inside. (ps aux shows just bash, init, and ps)

I thought this might be related to the number of open files, since a centos container will sometimes report "too many open files" when I run poweroff inside of it while I have a large number of LXC containers running. But I increased the file limits as described in this link, rebooted and verified the changes, but my problem persists.

What could be causing this?

  • 1
    You need to explain in the question how you know that that init is not systemd, and what program it instead is. – JdeBP Aug 28 '18 at 7:00
  • @JdeBP I guess I didn't know that init could be systemd. But when I run ps aux on the same container while everything works, I see a large number of other processes like systemd-journald, systemd-networkd, and importantly for my purposes, /usr/bin/sshd -D which is started by systemd. When things are not working, nothing related to systemd works, such as systemctl status sshd or poweroff. I just get this error: Failed to connect to bus: No such file or directory. – rexroni Aug 28 '18 at 13:39
  • Which Linux variant is your 20th container running, and how much system memory is available before after it starts? – Mark Stosberg Aug 28 '18 at 19:53
  • 1
    @MarkStosberg I confirmed the same behavior where the 20th container was Fedora, Arch, or Ubuntu, and sometimes (but not always) Debian. As far as memory goes, I am pretty sure I was running at around 15GB of memory available (out of 32GB, but no swap enabled). I can't check because today the problem seems to not want to reproduce. This has been bugging me for a couple weeks and when I got it to reproduce through multiple reboots I posted the question... but the issue seems to be not reproducing at the moment *deep, dejected sigh* – rexroni Aug 29 '18 at 2:57
  • OK, so the issue probably isn't with 20th container but something else that sometimes correlates with it. This is a pretty specific issue-- I would try an LXC-specific support channel like a mailing list, IRC channel, etc if available. – Mark Stosberg Aug 29 '18 at 14:30

If your problem happens with non-privileged (ie: using user namespaces) containers, rather than privileged (ie: root) containers, I believe this would be caused by a restricted limit on the use of the inotify interface for the same user running containers. Apparently systemd relies on inotify. The Debian buster LXC package includes a setting loosening the sysctl fs.inotify.max_user_instances in /etc/sysctl.d/30-lxc-inotify.conf:

# Defines the maximum number of inotify listeners.
# By default, this value is 128, which is quickly exhausted when using
# systemd-based LXC containers (15 containers are enough).
# When the limit is reached, systemd becomes mostly unusable, throwing
# "Too many open files" all around (both on the host and in containers).
# See https://kdecherf.com/blog/2015/09/12/systemd-and-the-fd-exhaustion/
# Increase the user inotify instance limit to allow for about
# 100 containers to run before the limit is hit again
fs.inotify.max_user_instances = 1024

The relevant documentation for this setting tells:

This specifies an upper limit on the number of inotify instances that can be created per real user ID.

If your LXC installation doesn't include such a setting, you could check if simply running on the host:

sysctl -w fs.inotify.max_user_instances=1024

will fix the problem.

| improve this answer | |
  • Neat! I am not currently in a position to reproduce the bug and check if this works, but I will come back and mark the answer as correct if it does. – rexroni Jun 3 '19 at 2:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.