Cgroup and process memory statistics mismatch

Question

I am trying to set up some monitoring to see when a service is using too much memory. The memory usage can be read from two places:

• the /proc/<pid>/status for the pid, or
• the /sys/fs/cgroup/<group-id>/memory.stat for the control group it runs in.

The service is started by systemd, so it gets its own control group, and because it sometimes starts child processes that I need to include in the statistics, and because the path is constant across restarts, the control group statistic is more appropriate.

Unfortunately the numbers don't seem to match. Here is a sample of the values at a point when no subprocesses are running (the command is exactly as executed except for the service name, and the result is exactly as obtain except non-memory-related items removed):

# cat /sys/fs/cgroup/system.slice/some.service/memory.stat /proc/$(cat /sys/fs/cgroup/system.slice/some.service/cgroup.procs)/status
anon 5873664
file 2408448
kernel_stack 491520
slab 962560
sock 0
shmem 61440
file_mapped 405504
file_dirty 0
file_writeback 0
inactive_anon 0
active_anon 5853184
inactive_file 1916928
active_file 360448
unevictable 0
slab_reclaimable 270336
slab_unreclaimable 692224
pgfault 60258
pgmajfault 99
pgrefill 0
pgscan 0
pgsteal 0
pgactivate 0
pgdeactivate 0
pglazyfree 0
pglazyfreed 0
workingset_refault 0
workingset_activate 0
workingset_nodereclaim 0
…
VmPeak:   494812 kB
VmSize:   494164 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:     25836 kB
VmRSS:     25484 kB
RssAnon:            5468 kB
RssFile:           20016 kB
RssShmem:              0 kB
VmData:   464776 kB
VmStk:       132 kB
VmExe:       180 kB
VmLib:     23940 kB
VmPTE:       156 kB
VmSwap:        0 kB
voluntary_ctxt_switches:        9
nonvoluntary_ctxt_switches:     620

I would consider the appropriate value to be VmRSS (= RssAnon + RssFile + RssShmem) from the process statistic. But while I would think that anon of the group should be RssAnon of the process and file of the group should be RssFile of the process, they don't match. While anon is 5736 KB, RssAnon is only 5468 KB, and for file the difference is even bigger, with file being only 2352 KB, but RssFile being 20016 KB, almost order of magnitude difference.

Also there is memory.current file with one value that approximately matches anon + file + kernel_stack + slab + sock + shmem, but I don't see any matching value in the process status.

So why are the numbers so different, and which are more indicative of how much memory pressure the application is exerting on the system?

Note: using cgroup2 on kernel 4.19.72 (slightly stale embedded BSP).

Answer 1

From the Control Group v2 guide:

A memory area is charged to the cgroup which instantiated it and stays charged to the cgroup until the area is released. Migrating a process to a different cgroup doesn’t move the memory usages that it instantiated while in the previous cgroup to the new cgroup.

A memory area may be used by processes belonging to different cgroups. To which cgroup the area will be charged is in-deterministic; however, over time, the memory area is likely to end up in a cgroup which has enough memory allowance to avoid high reclaim pressure.

So we'll start with the difference between file in cgroups and RssFile in /proc/<pid>/status:

The process may be have opened 20016 KB of files, but those files pages might have already existed in the memory cache before because other processes already opened them, and their respective cgroups charged them. So out of those 20016 KB, only 2352 KB where charged by the cgroup, and the rest belong to other cgroups (with the processes that loaded those files before).

Regarding the difference between anon in cgroups and RssAnon in /proc/<pid>/status, I don't have any good explanation.

Regarding memory.current, as far as I know all of the kernel internal structures (like kernel_stack and slab) are only visible in the cgroup, and you cannot see per-process stats of those numbers, since /proc/<pid>/status only shows user-space memory information.