Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am not able to understand why would kernel issue this oom killer when I see enough memory is available:

Also why are there so many kernel cache pages allocated? I say enough memory is available after looking at

Normal

DMA

Normal free lines

This is an embedded nand flash based device with 256 MB RAM

Kernel : 2.6.31

 myshellscript invoked oom-killer: gfp_mask=0xd0, order=2, oomkilladj=0 
 Backtrace: 
 [<c0106494>] (dump_backtrace+0x0/0x110) from [<c03641a0>] (dump_stack+0x18/0x1c) 
 r6:000000d0 r5:c9040c60 r4:00000002 r3:c0448690 
 [<c0364188>] (dump_stack+0x0/0x1c) from [<c015a314>] (oom_kill_process.clone.11+0x60/0x1b4) 
 [<c015a2b4>] (oom_kill_process.clone.11+0x0/0x1b4) from [<c015a738>] (__out_of_memory+0x154/0x178) 
 r8:c21e86e0 r7:001fb000 r6:00000002 r5:000000d0 r4:c9b6e000 
 [<c015a5e4>] (__out_of_memory+0x0/0x178) from [<c015a980>] (out_of_memory+0x68/0xa0) 
 [<c015a918>] (out_of_memory+0x0/0xa0) from [<c015d230>] (__alloc_pages_nodemask+0x42c/0x520) 
 r5:00000002 r4:000000d0 
 [<c015ce04>] (__alloc_pages_nodemask+0x0/0x520) from [<c015d388>] (__get_free_pages+0x18/0x44) 
 [<c015d370>] (__get_free_pages+0x0/0x44) from [<c0109418>] (get_pgd_slow+0x1c/0xe0) 
 [<c01093fc>] (get_pgd_slow+0x0/0xe0) from [<c0129ab0>] (mm_init.clone.43+0xb0/0xf0) 
 r7:c90858c0 r6:00000000 r5:c90858c0 r4:ce1a6680 
 [<c0129a00>] (mm_init.clone.43+0x0/0xf0) from [<c0129c40>] (mm_alloc+0x34/0x44) 
 r6:0009230c r5:c90858c0 r4:ce1a6680 r3:00000000 
 [<c0129c0c>] (mm_alloc+0x0/0x44) from [<c0180f70>] (bprm_mm_init+0x14/0x148) 
 r4:c5154000 r3:cd472564 
 [<c0180f5c>] (bprm_mm_init+0x0/0x148) from [<c01812d0>] (do_execve+0xa8/0x254) 
 [<c0181228>] (do_execve+0x0/0x254) from [<c0106000>] (sys_execve+0x3c/0x5c) 
 [<c0105fc4>] (sys_execve+0x0/0x5c) from [<c0102e80>] (ret_fast_syscall+0x0/0x2c) 
 r7:0000000b r6:0009230c r5:0009237c r4:000922fc 
 Mem-info: 
 DMA per-cpu: 
 CPU 0: hi: 18, btch: 3 usd: 0 
 Normal per-cpu: 
 CPU 0: hi: 42, btch: 7 usd: 0 
 Active_anon:28162 active_file:16 inactive_anon:18037 
 inactive_file:13 unevictable:0 dirty:0 writeback:0 unstable:0 
 free:9998 slab:2447 mapped:164 pagetables:701 bounce:0 
 DMA free:17128kB min:1560kB low:1948kB high:2340kB active_anon:51068kB inactive_anon:10320kB active_file:24kB inactive_file:0kB unevictable:0kB present:97536kB pages_scanned:0 all_unreclaimable? no 
 lowmem_reserve[]: 0 158 158 
 Normal free:22864kB min:2600kB low:3248kB high:3900kB active_anon:61580kB inactive_anon:61828kB active_file:40kB inactive_file:52kB unevictable:0kB present:162560kB pages_scanned:0 all_unreclaimable? no 
 lowmem_reserve[]: 0 0 0 
 DMA: 2358*4kB 912*8kB 25*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 17128kB 
 Normal: 4266*4kB 657*8kB 32*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 22864kB 
 26591 total pagecache pages 
 0 pages in swap cache 
 Swap cache stats: add 0, delete 0, find 0/0 
 Free swap = 0kB 
 Total swap = 0kB 
 65536 pages of RAM 
 10471 free pages 
 3967 reserved pages 
 2447 slab pages 
 892 shared page count 
 389 shared pages
 620 mapped shared page count
 177 mapped shared pages
 0 pages swap cached
 2481 dma reserved pages
 19892 total user pages
 20512 RSS sum by tasks
 20512 RSS sum by page stats
 164 user cache pages
 26427 kernel cache pages
share|improve this question
1  
Is this 32bit? How often does it happen? Is it easily reproducable? I would check with strace how much the process actually allocates. –  klapaucius Jul 27 '12 at 14:09

2 Answers 2

Edit: This answer is incorrect. Though still a possible cause for oom-killer to be invoked, it is not the cause in this specific case.


It looks like this is due to memory fragmentation.

From the output you provided, the highest order contiguous memory block you have is a 32kb block in the normal zone. This means that if anything tries to allocate a chunk of memory larger than 32kb, it will fail.
Normally this doesn't necessarily mean that the oom-killer will be invoked (otherwise an application could request a huge chunk of memory and thus trigger OOM), however this is the kernel that is trying to allocate memory, and so it's a bit more serious. In this exact case, it looks like the allocation was triggered by starting a new process, and the kernel was trying to allocate memory for that process.

The kernel automatically tries to compact (defragment) memory and get larger chunks of contiguous memory available, but some pages cannot be moved. And the longer the system is running, the more scattered these 'unmovable' pages become.

So basically, there's not much you can do. Really the only option is to kill off processes so those unmovable pages can be freed up.


As for what in that above output indicates the memory fragmentation, it's the following lines

 DMA: 2358*4kB 912*8kB 25*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 17128kB 
 Normal: 4266*4kB 657*8kB 32*16kB 1*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB = 22864kB

An entry like 32*16kb means there are 32 16kb contiguous chunks of memory free.

share|improve this answer
    
But, from the logs I am trying to allocate order 2 pages. And there are enough order 3 pages available. Why is oom killer still being invoked ? "gfp_mask=0xd0, order=2" –  abc Jul 24 '12 at 4:50
    
@abc you're right, I completely missed that. I've gone skimming through the kernel code (not 2.6.31 though, but a close version), and I'm not seeing anything jump out at me :-( –  Patrick Jul 24 '12 at 12:49
    
ok thanks for looking. –  abc Jul 24 '12 at 14:43

Sorry, this should be a comment on Patrick's answer - but I haven't amassed enough rep on unix.stackexchange to be able to comment on his answer.

We are dealing with an unknown application on an unknown embedded platform. Obviously, if we had more information on those two points, we might have a better chance of answering abc's question. It would also be useful to know exactly how much memory the script is attempting to acquire.


I think Patrick is correct - there is insufficient contiguous DMA to be able to allow the process to run. This could be for the following possible reasons:

  1. The embedded system may have a custom implementation of paging
  2. The embedded system might not have an MMU
  3. The script could be invoking a IO driver that access the DMA exactly
  4. The script could contain 3rd party programs that require contiguous memory

etc...

I believe that if you reduced the DMA memory fragmentation, the OOM killer would not be jumping. Easiest way to quickly test that is to restart the embedded device and see if the OOM-killer is still being called.

I shall now continue bumping around the internet looking for a lightweight, embedded friendly way of de-fragmenting one's memory without resetting their device.

This link may be of interest: http://bl0rg.krunch.be/oom-frag.html

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.