One of our servers suddenly was not responding properly. After forcing a reboot, we could access the log file. It is filled with what I believe are Kernel Oopses. The oops of the first command (plot, a user program) to trigger one is shown below. Following this one, other processes also trigger oopses, the first line however says

sshd: page allocation stalls for 57387ms, order:1, mode:0x16040c0(GFP_KERNEL|__GFP_COMP|__GFP_NOTRACK), nodemask=(null)

It's not only sshd, but also crond, cp, nrpe, automount and others. It seems as if anything could trigger an oops after the original one. They continue to trigger oopses all time time so that every few seconds, an oops is wriiten into the log file.

Can I find out what went wrong? Why did the oopses never stop? Since reboot, the server is running without any problems I can find, however the plot program hasn't been run so far (as I don't know which user executed it in the first place ...).

plot: page allocation failure: order:0, mode:0x1411100(GFP_NOIO|__GFP_COLD|__GFP_NORETRY|__GFP_NOMEMALLOC), nodemask=(null)
plot cpuset=/ mems_allowed=0-1
CPU: 40 PID: 1340614 Comm: plot Not tainted 4.13.0-1.el6.elrepo.x86_64 #1
Hardware name: Supermicro X10DRi/X10DRI-T, BIOS 2.1 09/13/2016
Call Trace:
? drain_all_pages+0x1b9/0x1f0
? ext4_map_blocks+0x325/0x540 [ext4]
? submit_bio+0x77/0x150
? __pagevec_lru_add_fn+0x16a/0x2f0
? ext4_mpage_readpages+0x504/0x910 [ext4]
? __radix_tree_create+0xaf/0x180
? workingset_update_node+0x19/0x70
? __radix_tree_lookup+0x92/0x110
? find_get_entry+0x43/0x140
? __add_to_page_cache_locked+0x124/0x1c0
? add_to_page_cache_lru+0x92/0x100
? add_page_wait_queue+0x82/0xa0
cachefiles_read_backing_file+0x101/0x720 [cachefiles]
? _ext4_get_block+0x74/0x120 [ext4]
? vfs_statfs+0x1b/0xc0
? ext4_get_block+0x16/0x20 [ext4]
? generic_block_bmap+0x3f/0x50
? mapping_tagged+0x12/0x20
cachefiles_read_or_alloc_pages+0x2e7/0x3a0 [cachefiles]
? kmem_cache_alloc_trace+0x1ea/0x230
? wake_up_bit+0x41/0x50
? fscache_run_op+0x54/0x90 [fscache]
? __nfs_readpages_from_fscache+0xf0/0xf0 [nfs]
? nfs_fh_get_context+0x11/0x20 [nfs]
? fscache_alloc_retrieval+0xc6/0xf0 [fscache]
__fscache_read_or_alloc_pages+0x210/0x300 [fscache]
__nfs_readpages_from_fscache+0xba/0xf0 [nfs]
nfs_readpages+0xad/0x190 [nfs]
? __alloc_pages_nodemask+0x1af/0x2b0
? __radix_tree_lookup+0x92/0x110
? filemap_map_pages+0x315/0x4d0
? change_pte_range+0x1c8/0x3a0
? flush_tlb_func_common+0xa2/0x150
? pagevec_lru_move_fn+0xc5/0xe0
? find_next_bit+0xb/0x10
? cpumask_any_but+0x27/0x40
? find_vma+0x20/0x80
? exit_to_usermode_loop+0x58/0xd0
RIP: 0033:0x43807a
RSP: 002b:00007ffcfbe17720 EFLAGS: 00010202
RAX: 000000004e90ea80 RBX: 0000000000000066 RCX: 0000000000000168
RDX: 000000000047c60b RSI: 0000000000000168 RDI: 000000004e90ea80
RBP: 00007ffcfbe17730 R08: 0000003aec78e188 R09: 0000000000000001
R10: 0000000000000010 R11: fffffffffffff000 R12: 0000000000000168
R13: 0000000000000065 R14: 0000000000000001 R15: 0000000000000001
active_anon:4147502 inactive_anon:476052 isolated_anon:0
active_file:2426 inactive_file:27709508 isolated_file:256
unevictable:0 dirty:7 writeback:0 unstable:0
slab_reclaimable:371384 slab_unreclaimable:65708
mapped:4313 shmem:525467 pagetables:18502 bounce:0
free:88956 free_pcp:21 free_cma:0
Node 0 active_anon:8812820kB inactive_anon:932012kB active_file:3872kB inactive_file:54201636kB unevictable:0kB isolated(anon):0kB isolated(file):768kB mapped:5208kB dirty:8kB writeback:0kB shmem:32kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 102400kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Node 1 active_anon:7777188kB inactive_anon:972196kB active_file:5832kB inactive_file:56636396kB unevictable:0kB isolated(anon):0kB isolated(file):256kB mapped:12044kB dirty:20kB writeback:0kB shmem:2101836kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 38912kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Node 0 DMA free:15884kB min:8kB low:20kB high:32kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15968kB managed:15884kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 1809 64269 64269
Node 0 DMA32 free:251108kB min:1264kB low:3116kB high:4968kB active_anon:433048kB inactive_anon:10372kB active_file:96kB inactive_file:1242292kB unevictable:0kB writepending:0kB present:1967292kB managed:1951368kB mlocked:0kB kernel_stack:20kB pagetables:664kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 0 62460 62460
Node 0 Normal free:43792kB min:43700kB low:107656kB high:171612kB active_anon:8379772kB inactive_anon:921640kB active_file:3728kB inactive_file:52958832kB unevictable:0kB writepending:8kB present:65011712kB managed:63959216kB mlocked:0kB kernel_stack:6116kB pagetables:28756kB bounce:0kB free_pcp:84kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 0 0 0
Node 1 Normal free:45040kB min:45132kB low:111184kB high:177236kB active_anon:7777188kB inactive_anon:972196kB active_file:5832kB inactive_file:56636416kB unevictable:0kB writepending:20kB present:67108864kB managed:66056324kB mlocked:0kB kernel_stack:8728kB pagetables:44588kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 1*4kB (U) 1*8kB (U) 0*16kB 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15884kB
Node 0 DMA32: 5882*4kB (UME) 5631*8kB (UME) 3549*16kB (UME) 2064*32kB (UME) 689*64kB (UME) 110*128kB (UME) 2*256kB (M) 0*512kB 1*1024kB (M) 0*2048kB 0*4096kB = 251120kB
Node 0 Normal: 344*4kB (UMEH) 2832*8kB (UMEH) 1133*16kB (UEH) 17*32kB (UEH) 7*64kB (H) 6*128kB (H) 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44176kB
Node 1 Normal: 945*4kB (UMEH) 3129*8kB (UMEH) 922*16kB (UMEH) 55*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45324kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
10421934 total pagecache pages
4010 pages in swap cache
Swap cache stats: add 2182308, delete 2178023, find 1238728/1661691
Free swap  = 44475900kB
Total swap = 45257724kB
33525959 pages RAM
0 pages HighMem/MovableOnly
530261 pages reserved
0 pages HighMem/MovableOnly
530261 pages reserved
0 pages hwpoisoned
  • 4
    This is an indication of failing hardware – jordanm Mar 13 '18 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.