2

in our production systems we are using Linux (kernel 5.10.55-051055-generic) on Intel NUCs (4GB RAM, 2GB swap) together with our Services run in Docker container. Those services are mainly small services communicating with zmq (over tcp) but also a service running CNNs using OpenVino on the integrated Intel GPU.

Over time (systems where we checked memory were running about 30 days) we somehow "lose" memory which means that a lot of memory (over 1 GB) is neither shown to be used by a process nor available. So the question is, what is using this memory or why is it not available?

What did we check:

  • docker stats memory usage (as we are running only docker container in an Ubuntu command line environment): However those values are not very useful for investigating this problem)
  • docker container memory usage shown in /sys/fs/cgroup/memory/docker/<container_id>/memory.usage_in_bytes and ...memory.stats: There is a large gap between memory used by the docker container and available memory
  • Log output of oom killer: Summing up the memory usage per process in the output sums up to only about 2.5GB, which means more than 1GB is somehow lost
  • Data from /proc/<process_id>/smaps (calculating ourself and using procrank): Sums up to only 2 to 2.5 GB when only a few hundred MB are available (again missing more than 1GB)
  • pidstat -trl: again missing more than 1GB
  • echo m > /proc/sysrq-trigger: again missing more than 1GB

Output of free command and procrank summary (created within 2 seconds):

                          Pss       Uss
                          1880676K  1855268K  TOTAL

RAM: 3672156K total, 136388K free, 17756K buffers, 417332K cached, 249268K shmem, 229920K slab

              total        used        free      shared  buff/cache   available
Mem:        3672156     3002328      138476      249268      531352      175376
Swap:       1951740     1951740           0

Output of echo m > /proc/sysrq-trigger:

[2948794.936393] sysrq: Show Memory
[2948794.936404] Mem-Info:
[2948794.936412] active_anon:196971 inactive_anon:206372 isolated_anon:0
                  active_file:109642 inactive_file:83546 isolated_file:0
                  unevictable:36 dirty:369 writeback:0
                  slab_reclaimable:27505 slab_unreclaimable:37597
                  mapped:84417 shmem:222 pagetables:6893 bounce:0
                  free:61015 free_pcp:1514 free_cma:0
[2948794.936417] Node 0 active_anon:787884kB inactive_anon:825488kB active_file:438568kB inactive_file:334184kB unevictable:144kB isolated(anon):0kB isolated(file):0kB mapped:337668kB dirty:1476kB writeback:0kB shmem:888kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:22176kB all_unreclaimable? no
[2948794.936419] Node 0 DMA free:14340kB min:296kB low:368kB high:440kB reserved_highatomic:0KB active_anon:68kB inactive_anon:144kB active_file:68kB inactive_file:140kB unevictable:0kB writepending:0kB present:15992kB managed:15904kB mlocked:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[2948794.936425] lowmem_reserve[]: 0 772 3487 3487 3487
[2948794.936431] Node 0 DMA32 free:96236kB min:14912kB low:18640kB high:22368kB reserved_highatomic:0KB active_anon:134108kB inactive_anon:163616kB active_file:40544kB inactive_file:80604kB unevictable:0kB writepending:20kB present:902612kB managed:837040kB mlocked:0kB pagetables:4820kB bounce:0kB free_pcp:1896kB local_pcp:444kB free_cma:0kB
[2948794.936439] lowmem_reserve[]: 0 0 2714 2714 2714
[2948794.936448] Node 0 Normal free:133484kB min:52368kB low:65460kB high:78552kB reserved_highatomic:2048KB active_anon:653708kB inactive_anon:661728kB active_file:397956kB inactive_file:253440kB unevictable:144kB writepending:1456kB present:2891776kB managed:2786464kB mlocked:48kB pagetables:22752kB bounce:0kB free_pcp:4160kB local_pcp:856kB free_cma:0kB
[2948794.936455] lowmem_reserve[]: 0 0 0 0 0
[2948794.936460] Node 0 DMA: 7*4kB (UME) 3*8kB (UE) 15*16kB (UME) 7*32kB (ME) 4*64kB (UM) 4*128kB (UME) 3*256kB (UME) 2*512kB (ME) 1*1024kB (E) 1*2048kB (E) 2*4096kB (M) = 14340kB
[2948794.936482] Node 0 DMA32: 3295*4kB (UME) 4936*8kB (UME) 1099*16kB (UME) 154*32kB (UME) 53*64kB (UME) 16*128kB (ME) 9*256kB (ME) 6*512kB (UM) 6*1024kB (ME) 2*2048kB (M) 0*4096kB = 96236kB
[2948794.936505] Node 0 Normal: 3621*4kB (MEH) 1357*8kB (MEH) 2857*16kB (UMEH) 693*32kB (UMEH) 259*64kB (UMEH) 41*128kB (ME) 22*256kB (ME) 11*512kB (ME) 7*1024kB (M) 0*2048kB 0*4096kB = 133484kB
[2948794.936526] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[2948794.936528] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[2948794.936529] 184470 total pagecache pages
[2948794.936532] 10069 pages in swap cache
[2948794.936534] Swap cache stats: add 1891574, delete 1881493, find 2324524/2691454
[2948794.936535] Free swap  = 832060kB
[2948794.936536] Total swap = 1951740kB
[2948794.936537] 952595 pages RAM
[2948794.936539] 0 pages HighMem/MovableOnly
[2948794.936541] 42743 pages reserved
[2948794.936542] 0 pages hwpoisoned

output of df -h | grep -e tmpfs -e Filesystem

Filesystem Size Used Avail Use% Mounted on 
tmpfs 359M 4.2M 355M 2% /run 
tmpfs 1.8G 0 1.8G 0% /dev/shm 
tmpfs 5.0M 0 5.0M 0% /run/lock 
tmpfs 1.8G 0 1.8G 0% /sys/fs/cgroup 
tmpfs 359M 0 359M 0% /run/user/0 
tmpfs 359M 0 359M 0% /run/user/1000

Edit: htop output (only parent processes, sorted by mem%) enter image description here

8
  • Are you using ZFS?
    – muru
    Commented Nov 28, 2023 at 14:25
  • do you use a 64-bit linux kernel? and is the VM also 64-bit? Commented Nov 28, 2023 at 14:45
  • It's basically a standard Ubuntu Server 20.04 OS. ext4 filesystem, 64-bit linux, no VM, Docker version 20.10.7
    – ts_arivo
    Commented Nov 28, 2023 at 14:54
  • What's the output of df | grep tmpfs? Commented Nov 29, 2023 at 6:57
  • @ArtemS.Tashkinov df -h | grep -e tmpfs -e Filesystem Filesystem Size Used Avail Use% Mounted on tmpfs 359M 4.2M 355M 2% /run tmpfs 1.8G 0 1.8G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 1.8G 0 1.8G 0% /sys/fs/cgroup tmpfs 359M 0 359M 0% /run/user/0 tmpfs 359M 0 359M 0% /run/user/1000```
    – ts_arivo
    Commented Nov 29, 2023 at 8:29

1 Answer 1

0

I guess this is because RAM is consumed by docker itself to maintain the network stack etc. Also, some memory is just used as cache. You can distinct it using commands like htop to point out where is used the "missing" memory.

Edit : Using htop and setup with F2 key You can do this adding column to point out specific process usage process in percent :

htop setup

1
  • 1
    The free command shows used = 3002328KB, which is memory used excluding buffers/caches. So this should be memory used by processes + kernel memory (slab) + shared memory. Memory of processes = 1880676KB, 249268KB shared memory, 229920KB slab = 2359864KB. So 642464KB are "missing". My question is, what is using this memory ?
    – ts_arivo
    Commented Nov 29, 2023 at 8:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .