I am trying to set up some monitoring to see when a service is using too much memory. The memory usage can be read from two places:
/proc/<pid>/statusfor the pid, or
/sys/fs/cgroup/<group-id>/memory.statfor the control group it runs in.
The service is started by systemd, so it gets its own control group, and because it sometimes starts child processes that I need to include in the statistics, and because the path is constant across restarts, the control group statistic is more appropriate.
Unfortunately the numbers don't seem to match. Here is a sample of the values at a point when no subprocesses are running (the command is exactly as executed except for the service name, and the result is exactly as obtain except non-memory-related items removed):
# cat /sys/fs/cgroup/system.slice/some.service/memory.stat /proc/$(cat /sys/fs/cgroup/system.slice/some.service/cgroup.procs)/status anon 5873664 file 2408448 kernel_stack 491520 slab 962560 sock 0 shmem 61440 file_mapped 405504 file_dirty 0 file_writeback 0 inactive_anon 0 active_anon 5853184 inactive_file 1916928 active_file 360448 unevictable 0 slab_reclaimable 270336 slab_unreclaimable 692224 pgfault 60258 pgmajfault 99 pgrefill 0 pgscan 0 pgsteal 0 pgactivate 0 pgdeactivate 0 pglazyfree 0 pglazyfreed 0 workingset_refault 0 workingset_activate 0 workingset_nodereclaim 0 … VmPeak: 494812 kB VmSize: 494164 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 25836 kB VmRSS: 25484 kB RssAnon: 5468 kB RssFile: 20016 kB RssShmem: 0 kB VmData: 464776 kB VmStk: 132 kB VmExe: 180 kB VmLib: 23940 kB VmPTE: 156 kB VmSwap: 0 kB voluntary_ctxt_switches: 9 nonvoluntary_ctxt_switches: 620
I would consider the appropriate value to be
RssShmem) from the process statistic. But while I would think that
anon of the group should be
RssAnon of the process and
file of the group should be
RssFile of the process, they don't match. While
anon is 5736 KB,
RssAnon is only 5468 KB, and for file the difference is even bigger, with
file being only 2352 KB, but
RssFile being 20016 KB, almost order of magnitude difference.
Also there is
memory.current file with one value that approximately matches
shmem, but I don't see any matching value in the process status.
So why are the numbers so different, and which are more indicative of how much memory pressure the application is exerting on the system?
Note: using cgroup2 on kernel 4.19.72 (slightly stale embedded BSP).