We are building or running Docker containers in our Jenkins instances built on top of Centos7 within AWS EC2.
We have 2 instances of t2.medium boxes with 2 CPUs and 3.5 Gb of Available memory.
In once case we are building the containers in another we are just pulling them and running (different container).
We started to get errors
open /var/lib/docker/overlay/<sha>-init/merged/dev/console: cannot allocate memory
journalctl we get
page allocation failure: order:4
Running page cache dump resolves the issue for a while
echo 1 > /proc/sys/vm/drop_caches
So what I noticed that while running the docker task,
Dirty memory block spikes (as it should) and
Mapped jumps after it. However, the
DirectMap4k is relatively close to that jump.
cat /proc/meminfo | grep -P "(Dirty|Mapped|DirectMap4k)" Dirty: 104 kB Mapped: 45696 kB DirectMap4k: 100352 kB
cat /proc/meminfo | grep -P "(Dirty|Mapped|DirectMap4k)" Dirty: 72428 kB Mapped: 70192 kB DirectMap4k: 100352 kB
So this machine takes some time to start failing, whereas identical machine reports
DirectMap4k: 77824 kB and thus fails regularly (it also has to handle building more complex container), but
sysctl vm is identical.
The underlying problem that build/boot of the docker container throws out of memory error and the question is what needs to be tuned for the kernel to make it stable.
Client: Version: 17.06.0-ce API version: 1.30 Go version: go1.8.3 Git commit: 02c1d87 Built: Fri Jun 23 21:20:36 2017 OS/Arch: linux/amd64 Server: Version: 17.06.0-ce API version: 1.30 (minimum version 1.12) Go version: go1.8.3 Git commit: 02c1d87 Built: Fri Jun 23 21:21:56 2017 OS/Arch: linux/amd64 Experimental: false
vm.admin_reserve_kbytes = 8192 vm.block_dump = 0 vm.dirty_background_bytes = 0 vm.dirty_background_ratio = 10 vm.dirty_bytes = 0 vm.dirty_expire_centisecs = 3000 vm.dirty_ratio = 30 vm.dirty_writeback_centisecs = 500 vm.drop_caches = 1 vm.extfrag_threshold = 500 vm.hugepages_treat_as_movable = 0 vm.hugetlb_shm_group = 0 vm.laptop_mode = 0 vm.legacy_va_layout = 0 vm.lowmem_reserve_ratio = 256 256 32 vm.max_map_count = 65530 vm.memory_failure_early_kill = 0 vm.memory_failure_recovery = 1 vm.min_free_kbytes = 67584 vm.min_slab_ratio = 5 vm.min_unmapped_ratio = 1 vm.mmap_min_addr = 4096 vm.nr_hugepages = 0 vm.nr_hugepages_mempolicy = 0 vm.nr_overcommit_hugepages = 0 vm.nr_pdflush_threads = 0 vm.numa_zonelist_order = default vm.oom_dump_tasks = 1 vm.oom_kill_allocating_task = 0 vm.overcommit_kbytes = 0 vm.overcommit_memory = 0 vm.overcommit_ratio = 50 vm.page-cluster = 3 vm.panic_on_oom = 0 vm.percpu_pagelist_fraction = 0 vm.stat_interval = 1 vm.swappiness = 30 vm.user_reserve_kbytes = 108990 vm.vfs_cache_pressure = 100 vm.zone_reclaim_mode = 0