I asked this question:
Running `sudo /sbin/service mysqld start` causes system to crash
In the comments of one of the answers, a very helpful individual is asking for more information regarding how much memory processes were using when oom-killer
is invoked.
The output of dmesg
is ~9000 lines, and hundreds of thousands of characters, though.
An example of the output that I see multiple times throughout the output is this:
Out of memory: Killed process 21000, UID 48, (httpd).
mysqld invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Call Trace:
[<ffffffff802c1b64>] out_of_memory+0x8b/0x203
[<ffffffff8020fa5d>] __alloc_pages+0x27f/0x308
[<ffffffff802139dd>] __do_page_cache_readahead+0xc8/0x1af
[<ffffffff8021424e>] filemap_nopage+0x14c/0x360
[<ffffffff80208e9d>] __handle_mm_fault+0x444/0x144f
[<ffffffff8020622a>] hypercall_page+0x22a/0x1000
[<ffffffff8020622a>] hypercall_page+0x22a/0x1000
[<ffffffff80266d94>] do_page_fault+0xf72/0x131b
[<ffffffff802456a8>] sys_rt_sigreturn+0x327/0x35a
[<ffffffff8026393d>] _spin_lock_irq+0x9/0x14
[<ffffffff802296ed>] do_sigaction+0x18c/0x1a1
[<ffffffff8025f82b>] error_exit+0x0/0x6e
Mem-info:
DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: high 186, batch 31 used:30
cpu 0 cold: high 62, batch 15 used:51
Normal per-cpu: empty
HighMem per-cpu: empty
Free pages: 4748kB (0kB HighMem)
Active:114975 inactive:0 dirty:0 writeback:0 unstable:0 free:1187 slab:4839 mapped-file:541 mapped-anon:114505 pagetables:1332
DMA free:2004kB min:48kB low:60kB high:72kB active:540kB inactive:0kB present:9076kB pages_scanned:5699645 all_unreclaimable? yes
lowmem_reserve[]: 0 489 489 489
DMA32 free:2744kB min:2804kB low:3504kB high:4204kB active:459360kB inactive:0kB present:500960kB pages_scanned:11998796 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2004kB
DMA32: 12*4kB 3*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2744kB
Normal: empty
HighMem: empty
586 pagecache pages
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap = 0kB
Total swap = 0kB
Free swap: 0kB
131072 pages of RAM
5993 reserved pages
5581 pages shared
0 pages swap cached
klogd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Call Trace:
[<ffffffff802c1b64>] out_of_memory+0x8b/0x203
[<ffffffff8020fa5d>] __alloc_pages+0x27f/0x308
[<ffffffff802139dd>] __do_page_cache_readahead+0xc8/0x1af
[<ffffffff8021424e>] filemap_nopage+0x14c/0x360
[<ffffffff80208e9d>] __handle_mm_fault+0x444/0x144f
[<ffffffff80263929>] _spin_lock_irqsave+0x9/0x14
[<ffffffff80263929>] _spin_lock_irqsave+0x9/0x14
[<ffffffff80251144>] finish_wait+0x32/0x5d
[<ffffffff80266d94>] do_page_fault+0xf72/0x131b
[<ffffffff802ff918>] kmsg_read+0x3a/0x44
[<ffffffff8025f82b>] error_exit+0x0/0x6e
Mem-info:
DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: high 186, batch 31 used:45
cpu 0 cold: high 62, batch 15 used:51
Normal per-cpu: empty
HighMem per-cpu: empty
Free pages: 4748kB (0kB HighMem)
Active:114975 inactive:0 dirty:0 writeback:0 unstable:0 free:1187 slab:4824 mapped-file:541 mapped-anon:114505 pagetables:1332
DMA free:2004kB min:48kB low:60kB high:72kB active:540kB inactive:0kB present:9076kB pages_scanned:6531037 all_unreclaimable? yes
lowmem_reserve[]: 0 489 489 489
DMA32 free:2744kB min:2804kB low:3504kB high:4204kB active:459360kB inactive:0kB present:500960kB pages_scanned:36231106 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2004kB
DMA32: 12*4kB 3*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2744kB
Normal: empty
HighMem: empty
586 pagecache pages
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap = 0kB
Total swap = 0kB
Free swap: 0kB
131072 pages of RAM
5993 reserved pages
5581 pages shared
0 pages swap cached
Out of memory: Killed process 21001, UID 48, (httpd).
sudo invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0
Call Trace:
[<ffffffff802c1b64>] out_of_memory+0x8b/0x203
[<ffffffff8020fa5d>] __alloc_pages+0x27f/0x308
[<ffffffff802139dd>] __do_page_cache_readahead+0xc8/0x1af
[<ffffffff8021424e>] filemap_nopage+0x14c/0x360
[<ffffffff80208e9d>] __handle_mm_fault+0x444/0x144f
[<ffffffff8020622a>] hypercall_page+0x22a/0x1000
[<ffffffff8020622a>] hypercall_page+0x22a/0x1000
[<ffffffff80266d94>] do_page_fault+0xf72/0x131b
[<ffffffff8024901b>] skb_dequeue+0x48/0x50
[<ffffffff80254146>] unix_release_sock+0x19e/0x1fa
[<ffffffff80261df5>] thread_return+0x6c/0x113
[<ffffffff80207116>] kmem_cache_free+0x84/0xd7
[<ffffffff80207116>] kmem_cache_free+0x84/0xd7
[<ffffffff8025f82b>] error_exit+0x0/0x6e
Mem-info:
DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
DMA32 per-cpu:
cpu 0 hot: high 186, batch 31 used:165
cpu 0 cold: high 62, batch 15 used:48
Normal per-cpu: empty
HighMem per-cpu: empty
Free pages: 4760kB (0kB HighMem)
Active:114996 inactive:0 dirty:0 writeback:0 unstable:0 free:1190 slab:4821 mapped-file:541 mapped-anon:114468 pagetables:1225
DMA free:2004kB min:48kB low:60kB high:72kB active:540kB inactive:0kB present:9076kB pages_scanned:7319613 all_unreclaimable? yes
lowmem_reserve[]: 0 489 489 489
DMA32 free:2756kB min:2804kB low:3504kB high:4204kB active:459444kB inactive:0kB present:500960kB pages_scanned:13331345 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
Normal free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 1*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2004kB
DMA32: 5*4kB 8*8kB 1*16kB 1*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2756kB
Normal: empty
HighMem: empty
586 pagecache pages
Swap cache: add 0, delete 0, find 0/0, race 0+0
Free swap = 0kB
Total swap = 0kB
Free swap: 0kB
131072 pages of RAM
5993 reserved pages
4629 pages shared
How do I read this in a way that can help me perform this task:
In the process list that gets dumped into dmesg at the time of the OOM condition, one of the columns should include how much memory each process is using. ...put that output in your question...