Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
I created a Cgroup on my Linux device and noticed that the value of memory.usage_in_bytes (4325376) is bigger than the sum of RSS, CACHE and SWAP (4194304).
I read in a reliable documentation, the memory.usage_in_bytes doesn't show the exact value of memory (and swap) usage. If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP) value in memory.stat.
Does anyone have idea about this? Should I use the value of memory.usage_in_bytes?
usage_in_bytes
For efficiency, as other kernel components, memory cgroup uses some optimization to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the method and doesn't show 'exact' value of memory (and swap) usage, it's a fuzz value for efficient access. (Of course, when necessary, it's synchronized.) If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP) value in memory.stat(see 5.2).
If you are reading the accurate files repeatedly, and you have measured your system performance as being limited by cpu usage, you might consider sampling memory_usage instead.
I don't know how it works. However as an LWN.net reader, this sounds similar to a situation where there can be multiple counters which are kept local to where they are used, and where somehow reading usage_in_bytes does not force an immediate synchronization and summary of these counters.
Over the years, kernel developers have made increasing use of per-CPU data in an effort to minimize memory contention and its associated performance penalties. As a simple example, consider the disk operation statistics maintained by the block layer. Incrementing a global counter for every disk operation would cause the associated cache line to bounce continually between processors; disk operations are frequent enough that the performance cost would be measurable. So each CPU maintains its own set of counters locally; it never has to contend with any other CPU to increment one of those counters. When a total count is needed, all of the per-CPU counters are added up. Given that the counters are queried far more rarely than they are modified, storing them in per-CPU form yields a significant performance improvement.
-- https://lwn.net/Articles/258238/
If you have a small system - not a large number of cpus, and not maintaining a large number of cgroups - the overhead doesn't sound very significant. It doesn't cost very much to bounce a few cache lines between cpus. But the efficiency might be useful if you are Google and running thousands of containers on a system :), or if you have some system with thousands of cpus.
I think I am seeing inconsistency when viewing memory.usage_in_bytes when the process under the two scenarios.
Scenario1: Process created directly under a cgroup Memory usage shown looks correct.
# cgexec -g memory:mem128 sleep 30 &
[2] 18339
# cat /sys/fs/cgroup/memory/mem128/memory.usage_in_bytes
135168
Scenario2: Process created separately and then assigned to a cgroup. Memory usage shown is incorrect. 36864 seems incorrect. Sometimes is shows 0 as well. Why is this different than the scenario 1 shown above?
# sleep 90 &
[2] 18937
# cgclassify -g memory:/mem128 18937
# cat /sys/fs/cgroup/memory/mem128/memory.usage_in_bytes
36864
