Why, after unmounting cgroup v1, do I still have empty directories under /sys/fs/cgroup?

Question

I just unmounted cgroup version 1, leaving just a single cgroup2 mount on my system.

$ mount | grep -i cgroup
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,size=4096k,nr_inodes=1024,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)

I was under the impression everything in /sys/fs/cgroup that was not /sys/fs/cgroup/unified is an artifact of cgroup 1. How come these remain though after unmounting cgroup version 1?

$ ls -lh
drwxr-xr-x  2 root root 40 Dec 25 18:57 blkio
lrwxrwxrwx  1 root root 11 Dec 25 18:57 cpu -> cpu,cpuacct
lrwxrwxrwx  1 root root 11 Dec 25 18:57 cpuacct -> cpu,cpuacct
drwxr-xr-x  2 root root 40 Dec 25 18:57 cpu,cpuacct
drwxr-xr-x  2 root root 40 Dec 25 18:57 cpuset
drwxr-xr-x  2 root root 40 Dec 25 18:57 devices
drwxr-xr-x  2 root root 40 Dec 25 18:57 freezer
drwxr-xr-x  2 root root 40 Dec 25 18:57 memory
lrwxrwxrwx  1 root root 16 Dec 25 18:57 net_cls -> net_cls,net_prio
drwxr-xr-x  2 root root 40 Dec 25 18:57 net_cls,net_prio
lrwxrwxrwx  1 root root 16 Dec 25 18:57 net_prio -> net_cls,net_prio
drwxr-xr-x  2 root root 40 Dec 25 18:57 perf_event
drwxr-xr-x  2 root root 40 Dec 25 18:57 pids
drwxr-xr-x  2 root root 40 Dec 25 18:57 rdma
drwxr-xr-x  2 root root 40 Dec 25 18:57 systemd
dr-xr-xr-x 13 root root  0 Dec 26 21:37 unified

Are these remaining temp dirs that are not kernel interfaces?

$ find . | grep -v unified
./freezer
./cpuset
./cpu
./cpuacct
./cpu,cpuacct
./rdma
./perf_event
./blkio
./memory
./devices
./net_prio
./net_cls
./net_cls,net_prio
./pids
./systemd

How does these empty directories work with cgroups v1?

Answer 1

This is because you're in systemd's "hybrid" mode, which uses cgroup v2 for its own internal bookkeeping, but still uses cgroup v1 for resource control. From the cgroup delegation documentation:

systemd supports three different modes how cgroups are set up. Specifically:

1. Unified — this is the simplest mode, and exposes a pure cgroup v2 logic. In this mode /sys/fs/cgroup is the only mounted cgroup API file system and all available controllers are exclusively exposed through it.

2. Legacy — this is the traditional cgroup v1 mode. In this mode the various controllers each get their own cgroup file system mounted to /sys/fs/cgroup/<controller>/. On top of that systemd manages its own cgroup hierarchy for managing purposes as /sys/fs/cgroup/systemd/.

3. Hybrid — this is a hybrid between the unified and legacy mode. It's set up mostly like legacy, except that there's also an additional hierarchy /sys/fs/cgroup/unified/ that contains the cgroup v2 hierarchy. (Note that in this mode the unified hierarchy won't have controllers attached, the controllers are all mounted as separate hierarchies as in legacy mode, i.e. /sys/fs/cgroup/unified/ is purely and exclusively about core cgroup v2 functionality and not about resource management.) In this mode compatibility with cgroup v1 is retained while some cgroup v2 features are available too. This mode is a stopgap. Don't bother with this too much unless you have too much free time.

To say this clearly, legacy and hybrid modes have no future. If you develop software today and don't focus on the unified mode, then you are writing software for yesterday, not tomorrow. They are primarily supported for compatibility reasons and will not receive new features.

Simply deleting the resource directories will render systemd unable to perform most resource control directives, since it no longer has access to the directory hierarchy required to do that.

Instead, to boot systemd with "unified" mode, which uses cgroup v2 for both its own internal bookkeeping and for resource control, you can boot the system with systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all on the kernel command line (or just cgroup_no_v1=all in v240+, see this patch).

cgroup_no_v1=all tells the kernel to disable all of the legacy cgroup hierarchies, which makes sure nobody can grab them and hold them hostage, and systemd.unified_group_hierarchy=1 tells systemd to use the unified cgroup hierarchy, not the hybrid hierarchy which is configured by default by your distribution.

Once you've booted with that, you'll find everything is located below /sys/fs/cgroup as you want:

% ls -l /sys/fs/cgroup | head
total 0
drwxr-xr-x  2 root root 0 Feb 17 17:19 dev-hugepages.mount/
drwxr-xr-x  2 root root 0 Feb 17 17:19 dev-mqueue.mount/
drwxr-xr-x  2 root root 0 Feb 17 17:19 init.scope/
drwxr-xr-x  2 root root 0 Feb 17 17:19 sys-fs-fuse-connections.mount/
drwxr-xr-x  2 root root 0 Feb 17 17:19 sys-kernel-config.mount/
drwxr-xr-x  2 root root 0 Feb 17 17:19 sys-kernel-debug.mount/
drwxr-xr-x  2 root root 0 Feb 17 17:19 sys-kernel-tracing.mount/
drwxr-xr-x 21 root root 0 Feb 21 13:00 system.slice/
drwxr-xr-x  4 root root 0 Feb 17 17:19 user.slice/

Answer 2

It's actually very simple. All cgroup mounts must be done on top of a directory. Before you had,

cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)

When you unmounted all these, you left the underlying directories there. Alas, these can be removed by remounting /sys/fs/cgroup as rw and simply deleting them.

sudo mount -o remount,rw /sys/fs/cgroup
# Delete the symlinks
sudo find /sys/fs/cgroup -maxdepth 1 -type l -exec rm {} \;
# Delete the empty directories
sudo find /sys/fs/cgroup/ -links 2 -type d -not -path '/sys/fs/cgroup/unified/*' -exec rmdir -v {} \;
sudo mount -o remount,ro /sys/fs/cgroup

After which you should just see your beautiful and clean cgroup2 remaining,

$ ls /sys/fs/cgroup
unified

Answer 3

Maybe save you hours of checking here and there, this is how to do it for Ubuntu 20.04.

0. Verify systemd version: systemd --version this should output version 240+ and therefore we can use short version according to this post
1. Add this after the last line that starts with GRUB_CMDLINE_LINUX

GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} cgroup_no_v1=all" This in effect will append cgroup_no_v1=all to the boot parameters.

2. Save the file and do sudo update-grub
3. reboot the machine.
4. Do cat /proc/cmdline after reboot to verify. If you see cgroup_no_v1=all in the output, it means cgroup v2 is active.