I am trying to set up some monitoring to see when a service is using too much memory. The memory usage can be read from two places:
- the
/proc/<pid>/status
for the pid, or - the
/sys/fs/cgroup/<group-id>/memory.stat
for the control group it runs in.
The service is started by systemd, so it gets its own control group, and because it sometimes starts child processes that I need to include in the statistics, and because the path is constant across restarts, the control group statistic is more appropriate.
Unfortunately the numbers don't seem to match. Here is a sample of the values at a point when no subprocesses are running (the command is exactly as executed except for the service name, and the result is exactly as obtain except non-memory-related items removed):
# cat /sys/fs/cgroup/system.slice/some.service/memory.stat /proc/$(cat /sys/fs/cgroup/system.slice/some.service/cgroup.procs)/status
anon 5873664
file 2408448
kernel_stack 491520
slab 962560
sock 0
shmem 61440
file_mapped 405504
file_dirty 0
file_writeback 0
inactive_anon 0
active_anon 5853184
inactive_file 1916928
active_file 360448
unevictable 0
slab_reclaimable 270336
slab_unreclaimable 692224
pgfault 60258
pgmajfault 99
pgrefill 0
pgscan 0
pgsteal 0
pgactivate 0
pgdeactivate 0
pglazyfree 0
pglazyfreed 0
workingset_refault 0
workingset_activate 0
workingset_nodereclaim 0
…
VmPeak: 494812 kB
VmSize: 494164 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 25836 kB
VmRSS: 25484 kB
RssAnon: 5468 kB
RssFile: 20016 kB
RssShmem: 0 kB
VmData: 464776 kB
VmStk: 132 kB
VmExe: 180 kB
VmLib: 23940 kB
VmPTE: 156 kB
VmSwap: 0 kB
voluntary_ctxt_switches: 9
nonvoluntary_ctxt_switches: 620
I would consider the appropriate value to be VmRSS
(= RssAnon
+ RssFile
+ RssShmem
) from the process statistic. But while I would think that anon
of the group should be RssAnon
of the process and file
of the group should be RssFile
of the process, they don't match. While anon
is 5736 KB, RssAnon
is only 5468 KB, and for file the difference is even bigger, with file
being only 2352 KB, but RssFile
being 20016 KB, almost order of magnitude difference.
Also there is memory.current
file with one value that approximately matches anon
+ file
+ kernel_stack
+ slab
+ sock
+ shmem
, but I don't see any matching value in the process status.
So why are the numbers so different, and which are more indicative of how much memory pressure the application is exerting on the system?
Note: using cgroup2 on kernel 4.19.72 (slightly stale embedded BSP).
memory.current
and thememory.swap.current
values of the cgroup?memory.current
matches (not exactly, but almost) sum of the first six values, and there is nomemory.swap.current
, because the device does not have swap enabled.