16

I'm trying to limit the total resources accessible from docker (for example only 90% of the RAM and 1500% of the CPU). I cannot use CPU and RAM limit when I'm launching my containers, that's why I need to limit the total resources available for docker containers.

I have around 20 containers which can consume the maximum CPU and memory but not at the same time, so I cannot set the CPU and RAM limit, that's why I need to limit the total resource used by docker

First of all I've created a slice: I tried the instruction above, but impossible to limit both the RAM and the CPU usage

# /etc/systemd/system/docker_limit.slice
[Unit]
Description=Slice that limits docker resources
Before=slices.target

[Slice]
CPUAccounting=true
CPUQuota=700%
#Memory Management
MemoryAccounting=true
MemoryHigh=20G
MemoryMax=25G
MemoryMaxSwap=10G

And my daemon.json

{
  "insecure-registries" : [ "url1", "url2"],
  "cgroup-parent": "docker_limit.slice"
}

But when I try from a container:

stress --vm-bytes $(awk '/MemAvailable/{printf "%d\n", $2 * 0.9;}' < /proc/meminfo)k --vm-keep -m 1

I can see from docker stats it's using 111Go of Ram (full capacity of my server)

stress --cpu 16

I can see from docker stats it's using near 1600 % (full capacity of my server)

I think I've missed something but I don't know what

0
9

For other person who will need a complete answer to this question, I'm doing a full recap:

First of all I've created a slice called docker_limit:

  1. Create a file to /etc/systemd/system/docker_limit.slice

    [Unit]
    Description=Slice that limits docker resources
    Before=slices.target
    
    [Slice]
    CPUAccounting=true
    CPUQuota=700%
    #Memory Management
    MemoryAccounting=true
    MemoryLimit=25G
    
  2. Start unit: systemctl start docker_limit.slice

  3. Edit /etc/docker/daemon.json

    {
      "cgroup-parent": "/docker_limit.slice"
    }
    
  4. Restart Docker daemon: systemctl restart docker

In order to verify all works as expected: systemd-cgtop, you should see processes happening bellow: /docker_slice

credit to @rwos

2
  • This allowed me to stop Docker from crashing my system, but the services I started exited with error instead of limiting their size, I don't know if that is because of the service, docker or some system configuration yet. Apr 20 at 7:14
  • This works, and is more updated baykara.medium.com/… Aug 27 at 11:33
7

Prefix the cgroup_parent in /etc/docker/daemon.json with a /, so the cgroup name is absolute. Otherwise, docker will put the containers into a sub-cgroup of the daemon's cgroup:

{
    "cgroup-parent": "/docker_limit.slice"
}

Without the /, depending on your system, the containers might end up in /system.slice/containerd.service/docker_limit.slice or similar.

systemd-cgtop or systemd-cgls can be used to check where things ended up running.

3
  • Thank you very much ! That was what I've missed
    – Leletir
    Nov 7 '19 at 15:53
  • Is it possible to specify % based metric for the RAM, instead of an absolute value. I'd like to set the daemon to use no more than 80% of the machine it is running on, and I have various machines - I don't want to create different file for different machines.
    – LIvanov
    Jan 6 '20 at 13:32
  • @LIvanov not that I know of, no - see freedesktop.org/software/systemd/man/… for what's possible to set (depends on the systemd/kernel version a bit).
    – rwos
    Jan 7 '20 at 15:09
2

I have a VM with 1 CPU and 2GB RAM (swap disabled).
Therefore I created the slice with next parameters:

# cat /etc/systemd/system/docker_limit.slice
[Unit]
Description=Slice that limits docker resources
Before=slices.target

[Slice]
CPUAccounting=true
CPUQuota=90%
MemoryAccounting=true
MemoryHigh=1G
MemoryMax=1.2G

And I made literally the same things as you, except native cgroupdriver in daemon:

# cat /etc/docker/daemon.json 
{
    "exec-opts": ["native.cgroupdriver=systemd"],
    "cgroup-parent": "docker_limit.slice"
}

Then I ran docker container with:

docker run --name stress-test -d ubuntu:18.04 sleep 10000

And opened my VM in 2 windows. In the 1st window, I ran stress. In the 2nd window, I ran the docker stats command. CPU was around 90%+-(htop confirm this and I saw some spikes to up to 91%) and when RAM was close to 1.2G, then this process exited with the message:

# stress --cpu 1 --vm 1 --vm-bytes 1300M
stress: info: [334] dispatching hogs: 1 cpu, 0 io, 1 vm, 0 hdd
stress: FAIL: [334] (415) <-- worker 336 got signal 9
stress: WARN: [334] (417) now reaping child worker processes
stress: FAIL: [334] (451) failed run completed in 4s

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.