8

When creating a userns (unprivileged) LXC container on Ubuntu 14.04 with the following command line:

lxc-create -n test1 -t download -- -d $(lsb_release -si|tr 'A-Z' 'a-z') -r $(lsb_release -sc) -a $(dpkg --print-architecture)

and (without touching the created configuration file) then attempting to start it with:

lxc-start -n test1 -l DEBUG

it fails. The log file shows me:

lxc-start 1420149317.700 INFO     lxc_start_ui - using rcfile /home/user/.local/share/lxc/test1/config
lxc-start 1420149317.700 INFO     lxc_utils - XDG_RUNTIME_DIR isn't set in the environment.
lxc-start 1420149317.701 INFO     lxc_confile - read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 1420149317.701 INFO     lxc_confile - read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 1420149317.701 WARN     lxc_log - lxc_log_init called with log already initialized
lxc-start 1420149317.701 INFO     lxc_lsm - LSM security driver AppArmor
lxc-start 1420149317.701 INFO     lxc_utils - XDG_RUNTIME_DIR isn't set in the environment.
lxc-start 1420149317.702 DEBUG    lxc_conf - allocated pty '/dev/pts/2' (5/6)
lxc-start 1420149317.702 DEBUG    lxc_conf - allocated pty '/dev/pts/7' (7/8)
lxc-start 1420149317.702 DEBUG    lxc_conf - allocated pty '/dev/pts/8' (9/10)
lxc-start 1420149317.702 DEBUG    lxc_conf - allocated pty '/dev/pts/10' (11/12)
lxc-start 1420149317.702 INFO     lxc_conf - tty's configured
lxc-start 1420149317.702 DEBUG    lxc_start - sigchild handler set
lxc-start 1420149317.702 DEBUG    lxc_console - opening /dev/tty for console peer
lxc-start 1420149317.702 DEBUG    lxc_console - using '/dev/tty' as console
lxc-start 1420149317.702 DEBUG    lxc_console - 14946 got SIGWINCH fd 17
lxc-start 1420149317.702 DEBUG    lxc_console - set winsz dstfd:14 cols:118 rows:61
lxc-start 1420149317.905 INFO     lxc_start - 'test1' is initialized
lxc-start 1420149317.906 DEBUG    lxc_start - Not dropping cap_sys_boot or watching utmp
lxc-start 1420149317.906 INFO     lxc_start - Cloning a new user namespace
lxc-start 1420149317.906 INFO     lxc_cgroup - cgroup driver cgmanager initing for test1
lxc-start 1420149317.907 ERROR    lxc_cgmanager - call to cgmanager_create_sync failed: invalid request
lxc-start 1420149317.907 ERROR    lxc_cgmanager - Failed to create hugetlb:test1
lxc-start 1420149317.907 ERROR    lxc_cgmanager - Error creating cgroup hugetlb:test1
lxc-start 1420149317.907 INFO     lxc_cgmanager - cgroup removal attempt: hugetlb:test1 did not exist
lxc-start 1420149317.908 INFO     lxc_cgmanager - cgroup removal attempt: perf_event:test1 did not exist
lxc-start 1420149317.908 INFO     lxc_cgmanager - cgroup removal attempt: blkio:test1 did not exist
lxc-start 1420149317.908 INFO     lxc_cgmanager - cgroup removal attempt: freezer:test1 did not exist
lxc-start 1420149317.909 INFO     lxc_cgmanager - cgroup removal attempt: devices:test1 did not exist
lxc-start 1420149317.909 INFO     lxc_cgmanager - cgroup removal attempt: memory:test1 did not exist
lxc-start 1420149317.909 INFO     lxc_cgmanager - cgroup removal attempt: cpuacct:test1 did not exist
lxc-start 1420149317.909 INFO     lxc_cgmanager - cgroup removal attempt: cpu:test1 did not exist
lxc-start 1420149317.910 INFO     lxc_cgmanager - cgroup removal attempt: cpuset:test1 did not exist
lxc-start 1420149317.910 INFO     lxc_cgmanager - cgroup removal attempt: name=systemd:test1 did not exist
lxc-start 1420149317.910 ERROR    lxc_start - failed creating cgroups
lxc-start 1420149317.910 INFO     lxc_utils - XDG_RUNTIME_DIR isn't set in the environment.
lxc-start 1420149317.910 ERROR    lxc_start - failed to spawn 'test1'
lxc-start 1420149317.910 INFO     lxc_utils - XDG_RUNTIME_DIR isn't set in the environment.
lxc-start 1420149317.910 INFO     lxc_utils - XDG_RUNTIME_DIR isn't set in the environment.
lxc-start 1420149317.910 ERROR    lxc_start_ui - The container failed to start.
lxc-start 1420149317.910 ERROR    lxc_start_ui - Additional information can be obtained by setting the --logfile and --logpriority options.

Now I see two errors here, the latter probably being a result of the former, which is:

lxc_start - failed creating cgroups

However, I see /sys/fs/cgroup mounted:

$ mount|grep cgr
none on /sys/fs/cgroup type tmpfs (rw)

and cgmanager is installed:

$ dpkg -l|awk '$1 ~ /^ii$/ && /cgmanager/ {print $2 " " $3 " " $4}'
cgmanager 0.24-0ubuntu7 amd64
libcgmanager0:amd64 0.24-0ubuntu7 amd64

Note: My host defaults still to upstart.

In case there's any doubt, the kernel support cgroups:

$ grep CGROUP /boot/config-$(uname -r)
CONFIG_CGROUPS=y
# CONFIG_CGROUP_DEBUG is not set
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_HUGETLB=y
CONFIG_CGROUP_PERF=y
CONFIG_CGROUP_SCHED=y
CONFIG_BLK_CGROUP=y
# CONFIG_DEBUG_BLK_CGROUP is not set
CONFIG_NET_CLS_CGROUP=m
CONFIG_NETPRIO_CGROUP=m

Note: My host defaults still to upstart.

7

Turns out, surprise surprise, this is a Ubuntu-specific thing.


The cause

The problem: although the kernel has cgroups enabled (check with grep CGROUP /boot/config-$(uname -r)) and cgmanager is running, there is no cgroup specific to my user. You can check that with:

$ cat /proc/self/cgroup
11:hugetlb:/
10:perf_event:/
9:blkio:/
8:freezer:/
7:devices:/
6:memory:/
5:cpuacct:/
4:cpu:/
3:name=systemd:/
2:cpuset:/

if your UID is given in each of the relevant lines, it's alright, but if no cgroups have been defined there will only be a slash after the second colon on each line.

My problem was specific to starting an unprivileged container. I could start privileged containers just fine.

It turned out that my problem was closely related to this thread on the lxc-users mailing list.

Remedy

On Ubuntu 14.04 upstart is the default, as opposed to systemd. Hence certain components that would be installed on a systemd-based distro do not get installed by default.

There were two packages in addition to cgmanager which I had to install in order to get beyond the error shown in my question: cgroup-bin and libpam-systemd. Quite frankly I am not 100% certain that the former is strictly needed, so you could try to leave it out and comment here.

After the installation of the packages and a reboot, you should then see your UID (id -u, here 1000) in the output:

$ cat /proc/self/cgroup
11:hugetlb:/user/1000.user/1.session
10:perf_event:/user/1000.user/1.session
9:blkio:/user/1000.user/1.session
8:freezer:/user/1000.user/1.session
7:devices:/user/1000.user/1.session
6:memory:/user/1000.user/1.session
5:cpuacct:/user/1000.user/1.session
4:cpu:/user/1000.user/1.session
3:name=systemd:/user/1000.user/1.session
2:cpuset:/user/1000.user/1.session

After that, the error upon attempting to start the guest container becomes (trimmed for brevity):

lxc-start 1420160065.383 INFO     lxc_cgroup - cgroup driver cgmanager initing for test1
lxc-start 1420160065.419 ERROR    lxc_start - failed to create the configured network
lxc-start 1420160065.446 ERROR    lxc_start - failed to spawn 'test1'
lxc-start 1420160065.451 ERROR    lxc_start_ui - The container failed to start.

So still no success, but we're one step closer.

The above linked lxc-users thread points to /etc/systemd/logind.conf not mentioning three controllers: net_cls, net_prio and debug. For me only the last one was missing. After the change you'll have to re-login, though, as the the changes take effect upon creation of your login session.

This blog post by one of the authors of LXC gives the next step:

Your user, while it can create new user namespaces in which it’ll be uid 0 and will have some of root’s privileges against resources tied to that namespace will obviously not be granted any extra privilege on the host.

One such thing is creating new network devices on the host or changing bridge configuration. To workaround that, we wrote a tool called “lxc-user-nic” which is the only SETUID binary part of LXC 1.0 and which performs one simple task. It parses a configuration file and based on its content will create network devices for the user and bridge them. To prevent abuse, you can restrict the number of devices a user can request and to what bridge they may be added.

An example is my own /etc/lxc/lxc-usernet file:

stgraber veth lxcbr0 10

This declares that the user “stgraber” is allowed up to 10 veth type devices to be created and added to the bridge called lxcbr0.

Between what’s offered by the user namespace in the kernel and that setuid tool, we’ve got all that’s needed to run most distributions unprivileged.

If your user has sudo rights and you're using Bash, use this:

echo "$(whoami) veth lxcbr0 10"|sudo tee -a /etc/lxc/lxc-usernet

and make sure the type (veth) matches the one in the container config and the bridge (lxcbr0) is configured and up.

And now we get another set of errors:

lxc-start 1420192192.775 INFO     lxc_start - Cloning a new user namespace
lxc-start 1420192192.775 INFO     lxc_cgroup - cgroup driver cgmanager initing for test1
lxc-start 1420192192.923 NOTICE   lxc_start - switching to gid/uid 0 in new user namespace
lxc-start 1420192192.923 ERROR    lxc_start - Permission denied - could not access /home/user.  Please grant it 'x' access, or add an ACL for the container root.
lxc-start 1420192192.923 ERROR    lxc_sync - invalid sequence number 1. expected 2
lxc-start 1420192192.954 ERROR    lxc_start - failed to spawn 'test1'
lxc-start 1420192192.959 ERROR    lxc_start_ui - The container failed to start.

Brilliant, that can be fixed. Another lxc-users thread by the same protagonists as in the first thread paves the way.

For now a quick test sudo chmod -R o+X $HOME will have to do, but ACLs are a viable option here as well. YMMV.

  • I'm still stuck by the fact that if I want to run userns LXC container as another user, it fails. Creation works (with one warning: WARN: could not reopen tty: Permission denied). But starting sudo -H -i -u database lxc-start -n mysql -d fails like in your question. Same errors. However your fix does not work with sudo. If I do sudo -H -i -u database cat /proc/self/cgroup I get the exact same output as if I run it as my calling user. So obviously, when trying to start the container using sudo, it will try as the other user to write in my cgroup which fails... :-( Any insight? – Huygens Sep 18 '15 at 22:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.