I use unprivileged lxc containers in Arch Linux. Here are the basic system infos:

[chb@conventiont ~]$ uname -a
Linux conventiont 3.17.4-Chb #1 SMP PREEMPT Fri Nov 28 12:39:54 UTC 2014 x86_64 GNU/Linux

It's a custom/compiled kernel with user namespace enabled:

[chb@conventiont ~]$ lxc-checkconfig 
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled
Multiple /dev/pts instances: enabled

--- Control groups ---
Cgroup: enabled
Cgroup clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled
Macvlan: enabled
Vlan: enabled
File capabilities: enabled

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

[chb@conventiont ~]$ systemctl --version
systemd 217
+PAM -AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD +IDN 

Unfortunately, systemd does not play well with lxc currently. Especially setting up cgroups for a non-root user seems to be working not well or I am just too unfamiliar how to do this. lxc will only start a container in unprivileged mode when it can create the necessary cgroups in /sys/fs/cgroup/XXX/*. This however is not possible for lxc because systemd mounts the root cgroup hierarchy in /sys/fs/cgroup/*. A workaround seems to be to do the following:

for d in /sys/fs/cgroup/*; do
        f=$(basename $d)
        echo "looking at $f"
        if [ "$f" = "cpuset" ]; then
                echo 1 | sudo tee -a $d/cgroup.clone_children;
        elif [ "$f" = "memory" ]; then
                echo 1 | sudo tee -a $d/memory.use_hierarchy;
        fi
        sudo mkdir -p $d/$USER
        sudo chown -R $USER $d/$USER
        echo $$ > $d/$USER/tasks
done

This code creates the corresponding cgroup directories in the cgroup hierarchy for an unprivileged user. However, something which I don't understand happens. Before executing the aforementioned I will see this:

[chb@conventiont ~]$ cat /proc/self/cgroup 
8:blkio:/
7:net_cls:/
6:freezer:/
5:devices:/
4:memory:/
3:cpu,cpuacct:/
2:cpuset:/
1:name=systemd:/user.slice/user-1000.slice/session-c1.scope

After executing the aforementioned code I see in the shell I ran it in:

[chb@conventiont ~]$ cat /proc/self/cgroup 
8:blkio:/chb
7:net_cls:/chb
6:freezer:/chb
5:devices:/chb
4:memory:/chb
3:cpu,cpuacct:/chb
2:cpuset:/chb
1:name=systemd:/chb

But in any other shell I still see:

[chb@conventiont ~]$ cat /proc/self/cgroup 
8:blkio:/
7:net_cls:/
6:freezer:/
5:devices:/
4:memory:/
3:cpu,cpuacct:/
2:cpuset:/
1:name=systemd:/user.slice/user-1000.slice/session-c1.scope

Hence, I can start my unprivileged lxc container in the shell I executed the code mentioned above but not in any other.

  1. Can someone explain this behaviour?

  2. Has someone found a better way to set up the required cgroups with a current version of systemd (>= 217)?

up vote 12 down vote accepted

A better and safer solution is to install cgmanager and run it with systemctl start cgmanager (on a systemd-based distro). You can than have your root user, or if you have sudo rights on the host create cgroups for your unprivileged user in all controllers with:

sudo cgm create all $USER
sudo cgm chown all $USER $(id -u $USER) $(id -g $USER)

Once they have been created for your unprivileged user she/he can move processes he has access to into his cgroup for every controller by using:

cgm movepid all $USER $PPID

Safer, faster, more reliable than the shell script I posted.

Manual solution:

To answer 1.

for d in /sys/fs/cgroup/*; do
        f=$(basename $d)
        echo "looking at $f"
        if [ "$f" = "cpuset" ]; then
                echo 1 | sudo tee -a $d/cgroup.clone_children;
        elif [ "$f" = "memory" ]; then
                echo 1 | sudo tee -a $d/memory.use_hierarchy;
        fi
        sudo mkdir -p $d/$USER
        sudo chown -R $USER $d/$USER
        echo $$ > $d/$USER/tasks
done

I was ignorant about what was going on exactly when I wrote that script but reading this and experimenting a bit helped me to understand what is going on. What I am basically doing in this script is to create a new cgroup session for the current user which is what I already stated above. When I run these commands in the current shell or run them in a script and make it so that it gets evaluated in the current shell and not in a subshell (via . script The . is important for this to work!) is that I not just open a new session for user but add the current shell as a process that runs in this new cgroup. I can achieve the same effect by running the script in a subshell and then descend into the cgroup hierarchy in the chb subcgroup and use echo $$ > tasks to add the current shell to every member of the chb cgroup hierarchy.

Hence, when I run lxc in that current shell my container will also become a member of all the chb subcgroups that the current shell is a member of. That is to say my container inherits the cgroup status of my shell. This also explains why it doesn't work in any other shell that is not part of the current chb subcgroups.

I still pass at 2.. We'll probably need to wait either for a systemd update or further Kernel developments to make systemd adopt a consistent behaviour but I prefer the manual setup anyway as it forces you to understand what you're doing.

  • can you not just mount the cgroups dir somewhere else (honest question)? there was a good deal of controversy over linux cgroups and systemd last year when the cgroups maintainer apparently decided to give systemd by name and other similar unnamed apps authority over cgroups handling in userspace. not sure how it all turned out, but i know that it was pretty up in the air whether a user could do this at all a year ago. – mikeserv Dec 4 '14 at 15:57
  • I probably could do that but I would have to prevent systemd from mounting the cgroup root directory in the first place. Whenever I login to my machine systemd will mount the root cgroup root hierarchy under /sys/fs/cgroup and adds a user cgroup only under the systemd part of the root cgroup (You can see this above.). The difference between systemd based distros and not systemd based distros before they switch is that e.g. in Ubuntu cgroup management is not in the hands of the init daemon. – lord.garbage Dec 4 '14 at 16:15
  • It is instead handled by a program like e.g. cgmanager. Or you can do it by hand as suggested in the link to kernel.org I posted above. I currently do not have a deep enough understanding of systemd cgroup management to fiddle with it deeper than I do now. But hopefully this will soon change. – lord.garbage Dec 4 '14 at 16:17
  • 1
    True, I remember you stating that in a comment to an answer I gave a long time ago. I'll inquire... – lord.garbage Dec 4 '14 at 16:35
  • 1
    The trick basically is: sudo systemctl start cgmanager && sudo cgm create all $USER && sudo cgm chown all $USER $(id -u) $(id -g) && sudo cgm movepid all $USER $PPID. The last command needs to be run in the current shell in order to add it to the new cgroup for $USER. – lord.garbage Dec 13 '14 at 12:16

Actually in archlinux, this won't work with e.g. a unprivileged user (recommended when using unpriv. lxc containers). i.e. that user doesn't have sudo :)

Instead, define group in /etc/cgconfig.conf, activate cgconfig, cgrules (libcgroup in AUR), add cgrules as well, done.. unpriv. user will also have the same rights.

In systemd 218 (I don't know when , but seems one has to add two more conditions as they are not set when created from the cgconfig way):

cat /etc/cgconfig.conf

group lxcadmin {
perm {
    task {
        uid = lxcadmin;
        gid = lxcadmin;
    }
    admin {
        uid = lxcadmin;
        gid = lxcadmin;
    }
}
cpu { }
memory { memory.use_hierarchy = 1; }  
blkio { }
cpuacct { }
cpuset { 
    cgroup.clone_children = 1;
    cpuset.mems = 0;
    cpuset.cpus = 0-3; 
}
devices { }
freezer { }
hugetlb { }
net_cls { }
}

cat /etc/cgrules.conf
lxcadmin        *       lxcadmin/

Assuming namespace is compiled in kernel.

This is a template, cpus can be according to how many cores you have, mem can be set to some actual value, etc etc.

EDIT 2: Finally, in systemd, should you wish to use auto-start with such an unprivileged user, you can do:

cp /usr/lib/systemd/system/lxc{,admin}\@.service , then add User=lxcadmin

and enable it for lxcadmin's container called lolz systemctl enable lxcadmin@lolz.

  • Thank you @Anthon, I never can ever get the code formatting right in these websites ,x – Malina Salina Jan 28 '15 at 22:50
  • Thank you. Sorry for the late reply. Your first point, "Actually in archlinux, this won't work with e.g. a unprivileged user (recommended when using unpriv. lxc containers). i.e. that user doesn't have sudo :)" does not stand as you only need your root administrator to create and chown you into all cgroup controllers. This is perfectly fine and secure.movepid can be done without root rights and hence, the unpriv. user does not need any sudorights. (Btw, libcgroup is not supposed to be used anymore. RHEL and others have deprecated it.) – lord.garbage Jan 31 '15 at 14:44
  • @Brauner. How do you autostart at boot, your unprivileged user's containers then? Actually your solutions listed only worked (and implied) a sudo user. Mine did not. You asked how to fix it. Anyway, there has just been an update, and cgconfig now fails to start, as user.slices are added automatically, ahead of the cgconfig settings it seems. These are lacking any user permissions (possibly a regression bug, am looking into it now). I didn't say it was the best solution. It was the/a solution to your inquiry. :) But my containers aren't starting on boot now, grrr. – Malina Salina Jan 31 '15 at 15:48
  • The reason I listed systemctl enable lxcadmin@container was so root could decide to run a unpriv container on boot. If the user himself uses it in --user (land), it would only boot when he logs in, not very useful for a server. And a note on your comment. chowning a user into all controllers, allows that user to start moving pid's into host space, I believe, which is quite a security risk. – Malina Salina Jan 31 '15 at 15:52
  • Erm, that seemingly is what you were doing with your method initially listed I guess, but look at this, even if it's ubuntu systemd package bugs.launchpad.net/ubuntu/+source/systemd/+bug/1413927 But something was updated in past days changing the logic.. I am trying to track it down. – Malina Salina Jan 31 '15 at 15:58

So I ran into the same problem when trying to get LXC unpriviliged containers working on CentOS 7. I didn't want to use cgmanager because I don't like introducing any additional services if not absolutely required. What I ended up doing instead is patching systemd using some patches from the ubuntu package and one custom patch to expand the list of cgroup controllers. I have the sources needed to build an RPM on my GitHub account at https://github.com/CtrlC-Root/rpmdist. I also have patched versions of shadow-utils (for subuids and subgids) and pam (for loginuid). After I install these RPMs and configure a user to run unpriviliged containers (assign subuids & subgids, allocate veth pairs in lxc-usernet, create .config/lxc/default.conf, etc) I can run LXC unprivileged containers just fine.

EDIT: Another reason I didn't want to use cgmanager is because I didn't want my regular users to have to use sudo at all. Regular users should be able to log in and everything should "just work" out of the box.

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.