0

I have been getting random kswapd0, and OOM killers even though available RAM -100MB. Already gone through many other similar issues, but I could not get why OOM killer triggered in my case. Hoping someone with knowledge can share some insight and set the direction for me to look into.

EDIT: From top, I get this output when OOM killer triggered. Also, I am wondering why kswap triggered though ~100MB available? Our application needs only ~90 max, and already this has ~50MB allocated. So it was trying only ~40MB when this happened.

top - 09:19:06 up 23:57,  0 users,  load average: 4.50, 2.61, 1.87
Tasks: 101 total,   2 running,  99 sleeping,   0 stopped,   0 zombie
%Cpu(s):  7.1 us, 62.5 sy,  0.0 ni,  0.0 id, 25.9 wa,  0.0 hi,  4.5 si,  0.0 st
KiB Mem :   507008 total,    99320 free,   355096 used,    52592 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   102308 avail Mem 

top - 09:19:09 up 23:57,  0 users,  load average: 4.50, 2.61, 1.87
Tasks: 100 total,   1 running,  98 sleeping,   0 stopped,   1 zombie
%Cpu(s): 35.8 us, 45.4 sy,  0.0 ni,  0.0 id, 17.4 wa,  0.0 hi,  1.4 si,  0.0 st
KiB Mem :   507008 total,   162280 free,   288952 used,    55776 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   168376 avail Mem

Following is the back trace I have.

2022-07-29T09:19:09.117931Z,BTL200072600123,3,0,kernel:,[86254.933997] Out of memory: Kill process 25402 (application) score 181 or sacrifice child
2022-07-29T09:19:09.117941Z,BTL200072600123,3,0,kernel:,[86254.934006] Killed process 25402 (application) total-vm:159852kB, anon-rss:75664kB, file-rss:16020kB
2022-07-29T09:19:09.095963Z,BTL200072600123,4,1,kernel:, [86254.932989] acquisition invoked oom-killer: gfp_mask=0x2084d0, order=0, oom_score_adj=0
2022-07-29T09:19:09.096076Z,BTL200072600123,4,1,kernel:, [86254.933012] CPU: 0 PID: 17939 Comm: acquisition Tainted: G           O    4.1.46 #5
2022-07-29T09:19:09.096142Z,BTL200072600123,4,1,kernel:, [86254.933019] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
2022-07-29T09:19:09.096206Z,BTL200072600123,4,1,kernel:, [86254.933025] Backtrace:
2022-07-29T09:19:09.096270Z,BTL200072600123,4,1,kernel:, [86254.933054] [<800132e4>] (dump_backtrace) from [<80013500>] (show_stack+0x18/0x1c)
2022-07-29T09:19:09.096334Z,BTL200072600123,4,1,kernel:, [86254.933060]  r7:00000000 r6:80a83c70 r5:600f0113 r4:00000000
202
2022-07-29T09:19:09.098354Z,BTL200072600123,4,1,kernel:, [86254.933467] Mem-Info:
2022-07-29T09:19:09.098411Z,BTL200072600123,4,1,kernel:, [86254.933485] active_anon:50599 inactive_anon:6859 isolated_anon:0
2022-07-29T09:19:09.098472Z,BTL200072600123,4,1,kernel:, [86254.933485]  active_file:136 inactive_file:159 isolated_file:0
2022-07-29T09:19:09.098530Z,BTL200072600123,4,1,kernel:, [86254.933485]  unevictable:16 dirty:0 writeback:0 unstable:0
2022-07-29T09:19:09.098589Z,BTL200072600123,4,1,kernel:, [86254.933485]  slab_reclaimable:1089 slab_unreclaimable:2343
2022-07-29T09:19:09.098648Z,BTL200072600123,4,1,kernel:, [86254.933485]  mapped:5971 shmem:8154 pagetables:534 bounce:0
2022-07-29T09:19:09.098704Z,BTL200072600123,4,1,kernel:, [86254.933485]  free:23627 free_pcp:0 free_cma:23127
2022-07-29T09:19:09.098765Z,BTL200072600123,4,1,kernel:, [86254.933518] Normal free:94380kB min:1972kB low:2464kB high:2956kB active_anon:201792kB inactive_anon:27364kB active_file:476kB inactive_file:560kB unevictable:64kB isolated(anon):0kB isolated(file):0kB present:522240kB managed:505984kB mlocked:64kB dirty:0kB writeback:0kB mapped:23728kB shmem:32540kB slab_reclaimable:4356kB slab_unreclaimable:9372kB kernel_stack:1032kB pagetables:2136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:92508kB writeback_tmp:0kB pages_scanned:6648 all_unreclaimable? yes
2022-07-29T09:19:09.098829Z,BTL200072600123,4,1,kernel:, [86254.933523] lowmem_reserve[]: 0 8 8
2022-07-29T09:19:09.098890Z,BTL200072600123,4,1,kernel:, [86254.933549] HighMem free:128kB min:128kB low:128kB high:132kB active_anon:604kB inactive_anon:72kB active_file:68kB inactive_file:76kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1024kB managed:1024kB mlocked:0kB dirty:0kB writeback:0kB mapped:156kB shmem:76kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no
2022-07-29T09:19:09.098950Z,BTL200072600123,4,1,kernel:, [86254.933555] lowmem_reserve[]: 0 0 0
2022-07-29T09:19:09.099011Z,BTL200072600123,4,1,kernel:, [86254.933564] Normal: 268*4kB (UEMRC) 128*8kB (UERC) 80*16kB (UERC) 8*32kB (RC) 0*64kB 1*128kB (C) 0*256kB 1*512kB (C) 0*1024kB 4*2048kB (C) 20*4096kB (C) = 94384kB
2022-07-29T09:19:09.099068Z,BTL200072600123,4,1,kernel:, [86254.933608] HighMem: 32*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 128kB
2022-07-29T09:19:09.099126Z,BTL200072600123,4,1,kernel:, [86254.933638] 8449 total pagecache pages
2022-07-29T09:19:09.099183Z,BTL200072600123,4,1,kernel:, [86254.933646] 0 pages in swap cache
2022-07-29T09:19:09.099240Z,BTL200072600123,4,1,kernel:, [86254.933652] Swap cache stats: add 0, delete 0, find 0/0
2022-07-29T09:19:09.099297Z,BTL200072600123,4,1,kernel:, [86254.933656] Free swap  = 0kB
2022-07-29T09:19:09.099353Z,BTL200072600123,4,1,kernel:, [86254.933661] Total swap = 0kB
2022-07-29T09:19:09.099408Z,BTL200072600123,4,1,kernel:, [86254.933665] 130816 pages RAM
2022-07-29T09:19:09.099464Z,BTL200072600123,4,1,kernel:, [86254.933670] 256 pages HighMem/MovableOnly
2022-07-29T09:19:09.099521Z,BTL200072600123,4,1,kernel:, [86254.933675] 4294905824 pages reserved
2022-07-29T09:19:09.099578Z,BTL200072600123,4,1,kernel:, [86254.933680] 65536 pages cma reserved
2
  • Not that it is the best solution but you have no swap configured. Add in a little swap space and your application can continue but it will be slow. Then you can figure out how much memory you should have in your system to support the app and either configure the app to use less or configure the system to have more.
    – doneal24
    Aug 15, 2022 at 19:56
  • 1
    @doneal24 We do not have swap as its embedded device, so resources are limited and swapping to disk was too slow in our case. Do you think zram can work without slowing down too much if we take ~5% of RAM? Aug 16, 2022 at 7:30

2 Answers 2

1

Cursory adding up these numbers, everything seems to be in order: your running out of RAM, and the OOM killer kills:

2022-07-29T09:19:09.098704Z,BTL200072600123,4,1,kernel:, [86254.933485]  free:23627 free_pcp:0 free_cma:23127

That's 23627 kB of free RAM, your systemd.resource-control is set to MemoryLow of 24.6 MB:

… Normal free:94380kB min:1972kB low:2464kB

So, yeah.

I'm getting random kswap0

um, you have 0 free swap:

Free swap  = 0kB
3
  • Muller But the Normal zone Free was 94380kB. Even whem OOM killer triggered, top output was showing ~90MB available. I have added this information to the first post, please check. How free:23627? what is the value ? Please check, and share your observations. Aug 16, 2022 at 7:46
  • That's directly read from the kernel logs you've shared. Can't trust top, it might not be updating as fast as something is eating up your RAM. Aug 16, 2022 at 8:06
  • How to read that value, please? I want to monitor that free memory reported by kernel. Thanks Aug 16, 2022 at 12:04
1

Linux does not randomly kill a process with an OOM unless said process is not able to allocate more RAM for the work it's expected to do.

From the numbers (anon-rss:75664kB), it looks like that one application is using a lot of memory and that's why it gets killed.

Two additional notes:

  1. You do not have any swap memory (Total swap = 0kB)

    This is great on a fast server which doesn't have apps that can run out of memory... otherwise, it's a good idea to add some swap, just in case. On my company's server, I have enough memory 99.9% of the time. Once in while, though, something starts running in the background and the server runs out of memory for a short time (until that one process quits). Having swap in that situation saves the day 99.9% of the time and it is not going to read/write like crazy because it's just a one time event.

  2. Which process gets killed is a complex heuristic.

    It is not automatically the one process just trying to allocate more memory. It is more likely the one that allocate the most memory. I do not know the algorithm exactly, but the OS will kick one process to have a greater chance that all the other processes will run smoothly. So process A may call malloc() and process B gets OOM.


Another note about large server: they may manage memory in "files" (separate "sections"). My server has 512Gb of RAM in two files. I can have a process run out of memory if it tries to use over 200Gb of RAM (which should not happen too often). If you also have two files, then a process is probably limited to around 64Gb.

3
  • Hi! Thanks for your answer. We have 512MB RAM as its embedded device, and eMMC which is not faster enough. How much swap you can recommend? As we have limited resources for both NAND and RAM. Is it good having ZRAM with ~5% of RAM? Aug 16, 2022 at 7:50
  • Embedded software should allocate memory that it needs on startup and then use those buffers for the rest of the time it runs. I know it's not always that easy to implement (I've done work on several embedded systems and modern systems are just too complex for that basic mechanism, in most cases). However, not having swap is a good idea for embedded software, so if you are in control of that software, I'd instead work on it to see what allocates memory and make sure it's not a memory leak. (you could use the sanatizer with gcc/g++). Aug 17, 2022 at 15:49
  • In the old days, the swap was set at a number about equal to 2x the RAM. That works when your memory is smaller (up to about 4Gb of RAM). Then you don't grow the swap memory much after that, because 8Gb of swap is going to be slow already and if you have 512Gb like me, you don't want a 1Tb drive just for swap (it would be DEAD slow). I have a swap space of 30Gb in my case. It never used much more than 2Gb though. (I think one time it used 8Gb as I was running a process leaking memory all night!) Aug 17, 2022 at 15:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .