6

I got the OOM problem randomly (about once per month), but it seems that there is enough free memory.

I can not understand the reason and how to resolve it ....

vm.min_free_kbytes = 90112

Linux xxx-alpha 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 GNU/Linux

Memory Monitor

slabtop

 Active / Total Objects (% used)    : 45902519 / 54189870 (84.7%)
 Active / Total Slabs (% used)      : 1260240 / 1260240 (100.0%)
 Active / Total Caches (% used)     : 151 / 187 (80.7%)
 Active / Total Size (% used)       : 9241442.17K / 10856556.48K (85.1%)
 Minimum / Average / Maximum Object : 0.01K / 0.20K / 23.25K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
43615236 37957660  87%    0.19K 1038467       42   8307736K dentry
2180640 1579066  72%    0.07K  38940       56    155760K range_seg_cache
706048 538395  76%    0.01K   1379      512      5516K kmalloc-8
644992 395397  61%    0.03K   5039      128     20156K kmalloc-32
635150 294932  46%    0.32K  12963       49    207408K arc_buf_hdr_t_full
546840 332510  60%    0.09K  13020       42     52080K kmalloc-96
dmesg

[Sun Jan 10 00:05:35 2021] elasticsearch[7 invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
[Sun Jan 10 00:05:35 2021] elasticsearch[7 cpuset=67e9bddb581923bf7602fbcce820867ff13dd19355d458112bee6300e0a2cbd8 mems_allowed=0-1
[Sun Jan 10 00:05:35 2021] CPU: 45 PID: 9599 Comm: elasticsearch[7 Tainted: P           OE     4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
[Sun Jan 10 00:05:35 2021] Hardware name: Ericsson CRU 0101/S2BS-MB, BIOS S2B_3B20 05/30/2019
[Sun Jan 10 00:05:35 2021] Call Trace:
[Sun Jan 10 00:05:35 2021]  dump_stack+0x5c/0x80
[Sun Jan 10 00:05:35 2021]  dump_header+0x6b/0x283
[Sun Jan 10 00:05:35 2021]  ? do_try_to_free_pages+0x2ec/0x370
[Sun Jan 10 00:05:35 2021]  oom_kill_process.cold.30+0xb/0x1cf
[Sun Jan 10 00:05:35 2021]  ? oom_badness+0x23/0x140
[Sun Jan 10 00:05:35 2021]  out_of_memory+0x1a5/0x430
[Sun Jan 10 00:05:35 2021]  __alloc_pages_slowpath+0xbd8/0xcb0
[Sun Jan 10 00:05:35 2021]  __alloc_pages_nodemask+0x28b/0x2b0
[Sun Jan 10 00:05:35 2021]  filemap_fault+0x3bd/0x780
[Sun Jan 10 00:05:35 2021]  ? filemap_map_pages+0x1ed/0x3a0
[Sun Jan 10 00:05:35 2021]  __do_fault+0x36/0x130
[Sun Jan 10 00:05:35 2021]  __handle_mm_fault+0xe6c/0x1270
[Sun Jan 10 00:05:35 2021]  handle_mm_fault+0xd6/0x200
[Sun Jan 10 00:05:35 2021]  __do_page_fault+0x249/0x4f0
[Sun Jan 10 00:05:35 2021]  ? page_fault+0x8/0x30
[Sun Jan 10 00:05:35 2021]  page_fault+0x1e/0x30
[Sun Jan 10 00:05:35 2021] RIP: 0033:0x7f68612bc435
[Sun Jan 10 00:05:35 2021] Code: 1f 00 00 49 c1 e0 03 41 8b 48 18 45 8b 48 1c 41 3b c9 0f 8d 47 21 00 00 4d 8b 50 10 8b d9 ff c3 41 89 58 18 49 8b fa 4c 63 d1 <46> 0f be 14 17 45 85 d2 0f 8c 07 0d 00 00 41 8b ca 83 e1 01 41 88
[Sun Jan 10 00:05:35 2021] RSP: 002b:00007e876f44e050 EFLAGS: 00010207
[Sun Jan 10 00:05:35 2021] RAX: 0000000000000002 RBX: 00000000000e37a6 RCX: 00000000000e37a5
[Sun Jan 10 00:05:35 2021] RDX: 0000000684825b90 RSI: 000000000032bce7 RDI: 00007f625cad5441
[Sun Jan 10 00:05:35 2021] RBP: 00000000d3887259 R08: 0000000684825bd0 R09: 00000000002f8d73
[Sun Jan 10 00:05:35 2021] R10: 00000000000e37a5 R11: 0000000684825b90 R12: 0000000000000000
[Sun Jan 10 00:05:35 2021] R13: 00000006829fb780 R14: 00000000d0904b72 R15: 00007f66dc05f800
[Sun Jan 10 00:05:35 2021] Mem-Info:
[Sun Jan 10 00:05:35 2021] active_anon:12218942 inactive_anon:93895 isolated_anon:0
                            active_file:1203 inactive_file:1345 isolated_file:6
                            unevictable:0 dirty:15 writeback:21 unstable:0
                            slab_reclaimable:1710574 slab_unreclaimable:1562977
                            mapped:26078 shmem:177912 pagetables:59851 bounce:0
                            free:49188 free_pcp:9835 free_cma:0
[Sun Jan 10 00:05:35 2021] Node 0 active_anon:31064964kB inactive_anon:48776kB active_file:284kB inactive_file:392kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:18116kB dirty:4kB writeback:4kB shmem:128568kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[Sun Jan 10 00:05:35 2021] Node 1 active_anon:17810804kB inactive_anon:326804kB active_file:4528kB inactive_file:4988kB unevictable:0kB isolated(anon):0kB isolated(file):24kB mapped:86196kB dirty:56kB writeback:80kB shmem:583080kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[Sun Jan 10 00:05:35 2021] Node 0 DMA free:15904kB min:20kB low:32kB high:44kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[Sun Jan 10 00:05:35 2021] lowmem_reserve[]: 0 1848 32052 32052 32052
[Sun Jan 10 00:05:35 2021] Node 0 DMA32 free:123288kB min:2556kB low:4420kB high:6284kB active_anon:1651524kB inactive_anon:2372kB active_file:116kB inactive_file:56kB unevictable:0kB writepending:0kB present:1959556kB managed:1892668kB mlocked:0kB kernel_stack:212kB pagetables:2500kB bounce:0kB free_pcp:248kB local_pcp:0kB free_cma:0kB
[Sun Jan 10 00:05:35 2021] lowmem_reserve[]: 0 0 30203 30203 30203
[Sun Jan 10 00:05:35 2021] Node 0 Normal free:22820kB min:42348kB low:73276kB high:104204kB active_anon:29413440kB inactive_anon:46404kB active_file:156kB inactive_file:336kB unevictable:0kB writepending:8kB present:31457280kB managed:30928584kB mlocked:0kB kernel_stack:21416kB pagetables:78456kB bounce:0kB free_pcp:6260kB local_pcp:0kB free_cma:0kB
[Sun Jan 10 00:05:35 2021] lowmem_reserve[]: 0 0 0 0 0
[Sun Jan 10 00:05:35 2021] Node 1 Normal free:34740kB min:45184kB low:78184kB high:111184kB active_anon:17810804kB inactive_anon:326804kB active_file:5176kB inactive_file:5044kB unevictable:0kB writepending:136kB present:33554432kB managed:33004524kB mlocked:0kB kernel_stack:47908kB pagetables:158448kB bounce:0kB free_pcp:32832kB local_pcp:1324kB free_cma:0kB
[Sun Jan 10 00:05:35 2021] lowmem_reserve[]: 0 0 0 0 0
[Sun Jan 10 00:05:35 2021] Node 0 DMA: 2*4kB (U) 1*8kB (U) 1*16kB (U) 2*32kB (U) 3*64kB (U) 0*128kB 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15904kB
[Sun Jan 10 00:05:35 2021] Node 0 DMA32: 99*4kB (UMH) 15*8kB (UMH) 3483*16kB (UMH) 2052*32kB (UMEH) 3*64kB (MH) 1*128kB (H) 3*256kB (H) 1*512kB (H) 0*1024kB 0*2048kB 0*4096kB = 123508kB
[Sun Jan 10 00:05:35 2021] Node 0 Normal: 882*4kB (UMEH) 33*8kB (MEH) 34*16kB (ME) 751*32kB (UME) 3*64kB (UH) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 28560kB
[Sun Jan 10 00:05:35 2021] Node 1 Normal: 8333*4kB (MH) 26*8kB (H) 1*16kB (H) 1*32kB (H) 3*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 33780kB
[Sun Jan 10 00:05:35 2021] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[Sun Jan 10 00:05:35 2021] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Sun Jan 10 00:05:35 2021] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[Sun Jan 10 00:05:35 2021] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Sun Jan 10 00:05:35 2021] 181392 total pagecache pages
[Sun Jan 10 00:05:35 2021] 0 pages in swap cache
[Sun Jan 10 00:05:35 2021] Swap cache stats: add 0, delete 0, find 0/0
[Sun Jan 10 00:05:35 2021] Free swap  = 0kB
[Sun Jan 10 00:05:35 2021] Total swap = 0kB
[Sun Jan 10 00:05:35 2021] 16746816 pages RAM
[Sun Jan 10 00:05:35 2021] 0 pages HighMem/MovableOnly
[Sun Jan 10 00:05:35 2021] 286396 pages reserved
[Sun Jan 10 00:05:35 2021] 0 pages hwpoisoned
[Sun Jan 10 00:05:35 2021] Tasks state (memory values in pages):
[Sun Jan 10 00:05:35 2021] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[Sun Jan 10 00:05:35 2021] [    611]     0   611    60450    40093   503808        0             0 systemd-journal
[Sun Jan 10 00:05:35 2021] [    635]     0   635     5616      348    61440        0         -1000 systemd-udevd
[Sun Jan 10 00:05:35 2021] [    814]     0   814      770       56    45056        0             0 mdadm
[Sun Jan 10 00:05:35 2021] [   9990]     0  9990    41684      221    69632        0             0 zed
[Sun Jan 10 00:05:35 2021] [   9991]     0  9991     2557       73    61440        0             0 cron
[Sun Jan 10 00:05:35 2021] [   9993]   104  9993     2222      187    57344        0          -900 dbus-daemon
[Sun Jan 10 00:05:35 2021] [   9994]     0  9994     4991      386    77824        0             0 systemd-logind
[Sun Jan 10 00:05:35 2021] [  10013]     0 10013   200898     2220    65536        0             0 supervisord
[Sun Jan 10 00:05:35 2021] [  10020]     0 10020    28023     1915   114688        0             0 unattended-upgr
[Sun Jan 10 00:05:35 2021] [  10028]     0 10028  1405091     6815   843776        0             0 containerd
[Sun Jan 10 00:05:35 2021] [  10033]     0 10033  1307161    15321   995328        0          -500 dockerd
[Sun Jan 10 00:05:35 2021] [  10040]     0 10040     3963      236    69632        0         -1000 sshd
[Sun Jan 10 00:05:35 2021] [  10074]     0 10074     4934     1407    73728        0             0 haproxy
[Sun Jan 10 00:05:35 2021] [  12823]     0 12823     1403       28    45056        0             0 agetty
[Sun Jan 10 00:05:35 2021] [  13945]   107 13945     4935      210    81920        0             0 exim4
[Sun Jan 10 00:05:35 2021] [  19622]     0 19622     5382      445    81920        0             0 systemd
[Sun Jan 10 00:05:35 2021] [  19623]     0 19623     5779      630    73728        0             0 (sd-pam)
[Sun Jan 10 00:05:35 2021] [  24193]     0 24193    31559     4572   143360        0             0 prom-node-expor
[Sun Jan 10 00:05:35 2021] [   9763]     0  9763    61147     1433    65536        0             0 nginx
[Sun Jan 10 00:05:35 2021] [   4864]     0  4864    58959      222    94208        0             0 polkitd
[Sun Jan 10 00:05:35 2021] [  19619]     0 19619   155613      481   126976        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  19633]     0 19633    27276      212    73728        0          -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [  19686]   472 19686    46091     7685   315392        0             0 grafana-server
[Sun Jan 10 00:05:35 2021] [  34921]     0 34921   820113      471   462848        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  34929]     0 34929    26924      202    73728        0          -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [  34948] 65534 34948  5391852    43302 21938176        0             0 prometheus
[Sun Jan 10 00:05:35 2021] [  12393]   108 12393    19145      168    65536        0             0 ntpd
[Sun Jan 10 00:05:35 2021] [   5862]     0  5862     4947     2284    90112        0             0 tmux: server
[Sun Jan 10 00:05:35 2021] [   5863]     0  5863     2209      151    61440        0             0 bash
[Sun Jan 10 00:05:35 2021] [  33048]     0 33048     2249      215    57344        0             0 bash
[Sun Jan 10 00:05:35 2021] [  29476]     0 29476     2201      154    53248        0             0 bash
[Sun Jan 10 00:05:35 2021] [  31493]     0 31493      597       46    45056        0             0 mysqld_safe
[Sun Jan 10 00:05:35 2021] [  45444]     0 45444     5315     1563    86016        0             0 mysql
[Sun Jan 10 00:05:35 2021] [  13606]     0 13606     2209      156    61440        0             0 bash
[Sun Jan 10 00:05:35 2021] [  35429]     0 35429     2179      152    49152        0             0 bash
[Sun Jan 10 00:05:35 2021] [  26987]     0 26987   137180      482   118784        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  27040]     0 27040   985866     1166   544768        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  27049]     0 27049    27276      940    86016        0          -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [  27074]     0 27074     4947       74    81920        0             0 wrapper
[Sun Jan 10 00:05:35 2021] [  28817]     0 28817     1098       22    53248        0             0 runsvdir
[Sun Jan 10 00:05:35 2021] [  29114]     0 29114     1060       19    49152        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  29117]     0 29117     1096       20    53248        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [  29120]     0 29120    16378      178   167936        0             0 sshd
[Sun Jan 10 00:05:35 2021] [  34657]     0 34657     1060       18    53248        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  34660]   997 34660    18749     2493   143360        0             0 redis-server
[Sun Jan 10 00:05:35 2021] [  35797]     0 35797     1096       22    57344        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [  37303]     0 37303     1060       18    49152        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  37305]   998 37305   760003      445   475136        0             0 gitaly-wrapper
[Sun Jan 10 00:05:35 2021] [  37311]   998 37311   759550    23379  1101824        0             0 gitaly
[Sun Jan 10 00:05:35 2021] [  37587]   998 37587   728803    17562   888832        0             0 ruby
[Sun Jan 10 00:05:35 2021] [  37588]   998 37588   778727    39586  1073152        0             0 ruby
[Sun Jan 10 00:05:35 2021] [  37595]     0 37595     1096       21    57344        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [  39290]     0 39290     1060       19    53248        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  39299]   996 39299    12338     2782   131072        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  39329]   996 39329    12392      920   110592        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  39330]   996 39330    12374      831   110592        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  39331]   996 39331    12338      543   102400        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  39332]   996 39332    12473     1007   118784        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  39333]   996 39333     8152      553    94208        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  39334]   996 39334    12451      709   118784        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  40268]     0 40268     1096       20    49152        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [  41373]     0 41373     1060       18    49152        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  41375]   998 41375   330686   207154  2560000        0             0 bundle
[Sun Jan 10 00:05:35 2021] [  42757]     0 42757     1096       22    53248        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [  43230]     0 43230     1060       19    45056        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  43232]   998 43232    34288     3288   274432        0             0 ruby
[Sun Jan 10 00:05:35 2021] [  43261]   998 43261   891610   247976  6795264        0             0 bundle
[Sun Jan 10 00:05:35 2021] [  44871]     0 44871     1096       21    53248        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [  45716]     0 45716     1060       18    49152        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  46557]     0 46557     1096       20    57344        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [  47976]     0 47976     1060       18    53248        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  47978]     0 47978    11158      287   114688        0             0 nginx
[Sun Jan 10 00:05:35 2021] [  48665]     0 48665     1096       20    57344        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [   1841]     0  1841     1060       18    53248        0             0 runsv
[Sun Jan 10 00:05:35 2021] [   2211]     0  2211     1096       20    53248        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [   4501]     0  4501     1060       18    53248        0             0 runsv
[Sun Jan 10 00:05:35 2021] [   4503]   998  4503   237878   114014  1216512        0             0 gitlab-exporter
[Sun Jan 10 00:05:35 2021] [   5326]     0  5326     1096       21    57344        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [   6712]     0  6712     1060       19    53248        0             0 runsv
[Sun Jan 10 00:05:35 2021] [   7193]     0  7193     1096       21    53248        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [   9114]     0  9114     1060       18    49152        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  11360]     0 11360     1096       23    57344        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [  13794]     0 13794     1060       19    53248        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  15522]     0 15522     1096       21    53248        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [  17228]     0 17228     1060       19    53248        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  18181]     0 18181     1096       22    53248        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [  20227]     0 20227     1060       18    49152        0             0 runsv
[Sun Jan 10 00:05:35 2021] [  21159]     0 21159     1096       21    53248        0             0 svlogd
[Sun Jan 10 00:05:35 2021] [  23043]   996 23043    15811     3066   151552        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  23048]   996 23048    15008     1972   143360        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  23052]   996 23052    15801     3037   151552        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  29245]   996 29245    14829     1534   143360        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  30217]   996 30217    14830     1535   143360        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  30363]   996 30363    15482     2791   147456        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  31925]   998 31925   727762     7223   606208        0             0 gitlab-workhors
[Sun Jan 10 00:05:35 2021] [  32066]   997 32066   640030     2595   475136        0             0 redis_exporter
[Sun Jan 10 00:05:35 2021] [  32276]   992 32276  1962869    75117  2211840        0             0 prometheus
[Sun Jan 10 00:05:35 2021] [  32438]   992 32438   696610     2725   503808        0             0 alertmanager
[Sun Jan 10 00:05:35 2021] [  32620]   996 32620   691549     4155   540672        0             0 postgres_export
[Sun Jan 10 00:05:35 2021] [  32626]   996 32626    21203     6899   180224        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  32693]   992 32693  1181528     5796   720896        0             0 grafana-server
[Sun Jan 10 00:05:35 2021] [  33814]     0 33814     4939       59    86016        0             0 gitlab-ctl
[Sun Jan 10 00:05:35 2021] [  33828]     0 33828    37816     6500   335872        0             0 omnibus-ctl
[Sun Jan 10 00:05:35 2021] [  34377]     0 34377     1126       18    57344        0             0 sh
[Sun Jan 10 00:05:35 2021] [  34380]     0 34380     1595       20    53248        0             0 xargs
[Sun Jan 10 00:05:35 2021] [  34381]     0 34381     1528       31    57344        0             0 tail
[Sun Jan 10 00:05:35 2021] [  35947]     0 35947     2179      145    49152        0             0 bash
[Sun Jan 10 00:05:35 2021] [  23562]   998 23562  9032150  7162562 62947328        0             0 mysqld
[Sun Jan 10 00:05:35 2021] [  44618]     0 44618   137180      481   122880        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  44627]     0 44627    27276      195    77824        0          -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [  44706]  2000 44706  9185087   552797  6643712        0             0 java
[Sun Jan 10 00:05:35 2021] [  41452]     0 41452   155613      480   126976        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  41460]     0 41460    27276      338    73728        0          -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [  41475]     0 41475    27276      255    73728        0          -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [  41494]   999 41494   327076    55842   933888        0             0 mongod
[Sun Jan 10 00:05:35 2021] [  41503]  1000 41503 422008290  1906347 96919552        0             0 java
[Sun Jan 10 00:05:35 2021] [  43137]     0 43137   137180      481   122880        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  43150]     0 43150   230017     1307   176128        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  43164]     0 43164   155613      479   131072        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  43178]     0 43178   137180      483   131072        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  43192]     0 43192   192959     1068   159744        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  43208]     0 43208   155613      478   135168        0          -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [  43234]     0 43234    27276      998    77824        0          -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [  43357]  1100 43357      571       20    45056        0             0 tini
[Sun Jan 10 00:05:35 2021] [  43798]  1100 43798     1372       78    49152        0             0 docker-entrypoi
[Sun Jan 10 00:05:35 2021] [  43828]  1100 43828 11417937  1054338 12394496        0             0 java
[Sun Jan 10 00:05:35 2021] [  23584]   106 23584     5178     1676    77824        0             0 haproxy
[Sun Jan 10 00:05:35 2021] [  23691]    33 23691    61703     1991    73728        0             0 nginx
[Sun Jan 10 00:05:35 2021] [  23692]    33 23692    61653     1844    73728        0             0 nginx
[Sun Jan 10 00:05:35 2021] [  24208]     0 24208   180714     7702   192512        0             0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [  24216]     0 24216   180650     6978   204800        0             0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [  24243]     0 24243   180842     7627   200704        0             0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [  24256]     0 24256   180906     7859   204800        0             0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [  24279]     0 24279   180778     7917   204800        0             0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [  24303]     0 24303   181146    10773   221184        0             0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [  48493]   998 48493   460417   215657  3211264        0             0 bundle
[Sun Jan 10 00:05:35 2021] [  48615]   996 48615    14829     1524   143360        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  20931]   998 20931   464001   225770  3223552        0             0 bundle
[Sun Jan 10 00:05:35 2021] [  20977]   996 20977    14829     1493   143360        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  46187]   998 46187   447326   220269  3133440        0             0 bundle
[Sun Jan 10 00:05:35 2021] [  46206]   996 46206    14829     1550   143360        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  27018]   998 27018   452737   226133  3174400        0             0 bundle
[Sun Jan 10 00:05:35 2021] [  27069]   996 27069    14829     1530   143360        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  10726]   996 10726    15611     3032   151552        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  43795]   996 43795    15497     2895   147456        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  39677]     0 39677     1126       17    49152        0             0 gitlab-logrotat
[Sun Jan 10 00:05:35 2021] [  46365]     0 46365     1094       17    53248        0             0 sleep
[Sun Jan 10 00:05:35 2021] [   4140]   996  4140    15522     3019   151552        0             0 postgres
[Sun Jan 10 00:05:35 2021] [  27524]     0 27524   156033      358   143360        0             0 rsyslogd
[Sun Jan 10 00:05:35 2021] [  23561]     0 23561     2785       98    57344        0             0 cron
[Sun Jan 10 00:05:35 2021] [  23572]     0 23572      597       16    45056        0             0 sh
[Sun Jan 10 00:05:35 2021] [  23573]     0 23573     2092       61    45056        0             0 ctl
[Sun Jan 10 00:05:35 2021] [  23839]     0 23839    18370     1260   131072        0             0 php
[Sun Jan 10 00:05:35 2021] [  24848]     0 24848     1358       42    53248        0             0 sshd
[Sun Jan 10 00:05:35 2021] Out of memory: Kill process 23562 (mysqld) score 436 or sacrifice child
[Sun Jan 10 00:05:35 2021] Killed process 23562 (mysqld) total-vm:36128600kB, anon-rss:28650288kB, file-rss:0kB, shmem-rss:0kB
[Sun Jan 10 00:05:37 2021] oom_reaper: reaped process 23562 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
3
  • There is a very similar answer: unix.stackexchange.com/questions/346208/… Jan 10, 2021 at 17:25
  • Do you have a periodic Raid volumes check scheduled monthly?
    – Marcelo
    Jan 10, 2021 at 17:29
  • @AlbertoPires thank you very much. I can understand the problem in the that answer now. If it is really the cause, what should we do? Should we limit the buff/cache memory of linux?
    – shawn
    Jan 10, 2021 at 18:20

2 Answers 2

2

There is an explanation at System unable to allocate memory even though memory is available.

As for how to solve the problem, I suggest to first, check to see if there are no memory leaks or unused processes that could be stopped, once you know you are running everything fine from the memory point of view, what's left is to increase physical memory and if that's not possible add some swap, as it can release some memory (may have a performance cost)

0

I have asked many people, but haven't got a good solution yet.

I highly suspect this problem is related to Linux kernel's dentries/inodes cache: when the free memory is insufficient, kernel doesn't try to reclaim the cache memory.

Every time I run echo 2 > /proc/sys/vm/drop_caches, the free memory becomes about 8G. Then it decreases to around 4G for about tens of minutes, then it decreases to around 2G, eventually it may become less than 1G, sometime less than 400M.

Only 2 for /proc/sys/vm/drop_caches affects the free memory, 1 has no effect.

I have to use a temporary solution (although I think it is bad ...), run echo 2 > /proc/sys/vm/drop_caches every 15 minutes ...... so sad.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .