5

we ran into an interesting bug today. on our servers we put users into cgroup folders to monitor + control usage of resources like cpu and memory. we started getting errors when trying to add user-specific memory cgroup folders:

mkdir /sys/fs/cgroup/memory/users/newuser
mkdir: cannot create directory ‘/sys/fs/cgroup/memory/users/newusers’: Cannot allocate memory

That seemed a little strange, because the machine actually had a reasonable amount of free memory and swap. Changing the sysctl values for vm.overcommit_memory from 0 to 1 had no effect.

We did notice that we were running with quite a lot of user-specific subfolders (about 7,000 in fact), and most of them were for users that were no longer running processes on that machine.

ls /sys/fs/cgroup/memory/users/ | wc -l
7298

deleting unused folders in the cgroup hierarchy actually fixed the problem

cd /sys/fs/cgroup/memory/users/
ls | xargs -n1 rmdir
# errors for folders in-use, succeeds for unused
mkdir /sys/fs/cgroup/memory/users/newuser
# now works fine

interestingly, the problem only affected the memory cgroup. the cpu/accounting cgroup was fine, even though it actually had more users in the hierarchy:

ls /sys/fs/cgroup/cpu,cpuacct/users/ | wc -l
7450
mkdir /sys/fs/cgroup/cpu,cpuacct/users/newuser
# fine

So, what was causing these out-of-memory errors? Does the memory-cgroup subsystem itself have some sort of memory limit of its own?

contents of cgroup mounts may be found here

0

There are indeed limits per cgroup, you can read about them on LWN.net:

Each cgroup has a memory controller specific data structure (mem_cgroup) associated with it.

.... Accounting happens per cgroup.

The maximum amount of memory is stored in /sys/fs/cgroup/memory/memory.limit_in_bytes. If the problem you experienced was really connected with cgroup memory limit, then /sys/fs/cgroup/memory/memory.max_usage_in_bytes should be close to the above, which you can also check by inspecting memory.failcnt, which records the number of times your actual usage hit the limit above.

Perhaps you may also check memory.kmem.failcnt and memory.kmem.tcp.failcnt for similar statistics on kernel memory and tcp buffer memory.

  • i'm not sure you've understood the question. the error i'm getting seems to be coming from the operating system when I'm trying to create a new cgroups folder -- it's not about the limits applied to any particular cgroup. correct me if i've misunderstood, myself... – hwjp Sep 20 '17 at 14:32
  • to answer your question, the top-level /sys/fs/cgroup/memory folder has the following: /sys/fs/cgroup/memory/memory.max_usage_in_bytes = 14010560512, /sys/fs/cgroup/memory/memory.limit_in_bytes = 9223372036854771712 (ie max usage 8 orders of magnitude under limit) – hwjp Sep 22 '17 at 9:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.