Why is machine.slice shown as using memory, when it does not contain any processes?

Question

$ systemctl status machine.slice
● machine.slice - Virtual Machine and Container Slice
   Loaded: loaded (/usr/lib/systemd/system/machine.slice; static; vendor preset: disabled)
   Active: active since Wed 2018-06-13 08:45:07 BST; 1 day 15h ago
     Docs: man:systemd.special(7)
    Tasks: 0
   Memory: 717.0M
   CGroup: /machine.slice

$ cd /sys/fs/cgroup/memory/machine.slice
$ cat memory.usage_in_bytes 
751915008
$ cat tasks
$ cat cgroup.procs
$

What does it mean, for the machine.slice cgroup to be using 717MB, even though it is empty of processes/threads? Is it a kernel bug?

I can reproduce this simply by starting a VM with virt-manager, and then stopping it. If I repeat the cycle, the result is about the same - i.e. the system doesn't appear to be leaking these hundreds of megabytes.

Software version

$ uname -r
4.16.14-300.fc28.x86_64

$ rpm -q systemd libvirt-daemon
systemd-238-8.git0e0aa59.fc28.x86_64
libvirt-daemon-4.1.0-2.fc28.x86_64

Answer 1

This is not a bug. Apparently the accounting of cgroup memory includes disk cache pages which were used by the processes. If desired, you can request to discard the cache pages which belong to a cgroup using force_empty.

You might wonder about cache pages which are used by multiple cgroups. cgroup-v2.rst (the successor to the version I am using) tells us:

A memory area is charged to the cgroup which instantiated it and stays charged to the cgroup until the area is released. Migrating a process to a different cgroup doesn't move the memory usages that it instantiated while in the previous cgroup to the new cgroup.

A memory area may be used by processes belonging to different cgroups. To which cgroup the area will be charged is in-deterministic; however, over time, the memory area is likely to end up in a cgroup which has enough memory allowance to avoid high reclaim pressure.

cgroup-v1/memory.txt also says that "pages are linked to per-memcg LRU exclusively", so it probably also works similar to the above description. This document is harder to rely on, as it starts with a disclaimer that it is "hopelessly outdated and it asks for a complete rewrite".

I.e. force_empty requests to drop all these pages immediately. If there is another cgroup that also wants to use them, it will have to read them in from disk again.