4

My CentOS 6.8 server is crashing every few days and I'm trying to figure out why. I'm using kernel version 2.6.32-642.1.1.el6.x86_64. So far, I've installed the program, crash, to help me sift through the kernel dump that was created from the kernel panic. But honestly, I don't really understand a lot of the info it outputs.

[root@resh boot]# crash /usr/lib/debug/lib/modules/2.6.32-642.1.1.el6.x86_64/vmlinux /var/crash/127.0.0.1-2016-08-02-09\:12\:20/vmcore
KERNEL: /usr/lib/debug/lib/modules/2.6.32-642.1.1.el6.x86_64/vmlinux
DUMPFILE: /var/crash/127.0.0.1-2016-08-02-09:12:20/vmcore  [PARTIAL DUMP]
CPUS: 32
DATE: Tue Aug  2 09:09:29 2016
UPTIME: 12:47:24
LOAD AVERAGE: 4.78, 4.66, 4.55
TASKS: 998
NODENAME: resh.cluster.org
RELEASE: 2.6.32-642.1.1.el6.x86_64
VERSION: #1 SMP Tue May 31 21:57:07 UTC 2016
MACHINE: x86_64  (2294 Mhz)
MEMORY: 31.8 GB
PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000002"
PID: 42993
COMMAND: "kslowd002"
TASK: ffff88040d88d520  [THREAD_INFO: ffff880100000000]
CPU: 7
STATE: TASK_RUNNING (PANIC)

crash backtrace:

crash> bt
PID: 42993  TASK: ffff88040d88d520  CPU: 7   COMMAND: "kslowd002"
#0 [ffff8801000039c0] machine_kexec at ffffffff8103fdcb
#1 [ffff880100003a20] crash_kexec at ffffffff810d1fe2
#2 [ffff880100003af0] oops_end at ffffffff8154bd00
#3 [ffff880100003b20] no_context at ffffffff810518cb
#4 [ffff880100003b70] __bad_area_nosemaphore at ffffffff81051b55
#5 [ffff880100003bc0] bad_area_nosemaphore at ffffffff81051c23
#6 [ffff880100003bd0] __do_page_fault at ffffffff8105231c
#7 [ffff880100003cf0] do_page_fault at ffffffff8154dc8e
#8 [ffff880100003d20] page_fault at ffffffff8154af95
[exception RIP: unknown or invalid address]
RIP: 0000000000000002  RSP: ffff880100003dd8  RFLAGS: 00010202
RAX: ffffffffa0465a80  RBX: ffff8801bc7da200  RCX: ffff8801bc7da2a8
RDX: 0000000000000002  RSI: 00000000ffffffff  RDI: ffff8801bc7da200
RBP: ffff880100003e20   R8: ffffffff81ad12d8   R9: fe2582cc8764a601
R10: 0000000000000001  R11: 0000000000000000  R12: 0000000000000000
R13: ffff8801bc7da248  R14: ffff8801bc7da290  R15: 00000000ffffffff
ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#9 [ffff880100003dd8] fscache_object_slow_work_execute at ffffffffa0460e9f [fscache]
#10 [ffff880100003e28] slow_work_execute at ffffffff81121363
#11 [ffff880100003e68] slow_work_thread at ffffffff81121645
#12 [ffff880100003ee8] kthread at ffffffff810a662e
#13 [ffff880100003f48] kernel_thread at ffffffff8100c28a

crash log:

BUG: unable to handle kernel NULL pointer dereference at 0000000000000002
IP: [<0000000000000002>] 0x2
PGD 0 
Oops: 0010 [#1] SMP 
last sysfs file: /sys/devices/system/cpu/online
CPU 7 
Modules linked in: nfs nfsd lockd nfs_acl auth_rpcgss sunrpc 8021q garp stp llc cpufreq_ondemand freq_table pcc_cpufreq cachefiles fscache(T) ipv6 ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables xfs exportfs ext2 power_meter acpi_ipmi ipmi_si ipmi_msghandler microcode iTCO_wdt iTCO_vendor_support hpilo hpwdt igb i2c_algo_bit i2c_core ptp pps_core sg serio_raw lpc_ich mfd_core ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif hpsa ahci dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 42993, comm: kslowd002 Tainted: G           -- ------------  T 2.6.32-642.1.1.el6.x86_64 #1 HP ProLiant DL360e Gen8
RIP: 0010:[<0000000000000002>]  [<0000000000000002>] 0x2
RSP: 0018:ffff880100003dd8  EFLAGS: 00010202
RAX: ffffffffa0465a80 RBX: ffff8801bc7da200 RCX: ffff8801bc7da2a8
RDX: 0000000000000002 RSI: 00000000ffffffff RDI: ffff8801bc7da200
RBP: ffff880100003e20 R08: ffffffff81ad12d8 R09: fe2582cc8764a601
R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000
R13: ffff8801bc7da248 R14: ffff8801bc7da290 R15: 00000000ffffffff
FS:  0000000000000000(0000) GS:ffff8800380e0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000002 CR3: 0000000001a8d000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kslowd002 (pid: 42993, threadinfo ffff880100000000, task ffff88040d88d520)
Stack:
 ffffffffa0460e9f ffff880100003e00 ffff8801bc7da238 ffff8801bc7da298
<d> 0000000000000002 ffffffff81f55ec0 ffff8801bc7da290 ffff8801bc7da298
<d> 0000000000000001 ffff880100003e60 ffffffff81121363 ffff880100003e60
Call Trace:
 [<ffffffffa0460e9f>] ? fscache_object_slow_work_execute+0xaf/0x1c0 [fscache]
 [<ffffffff81121363>] slow_work_execute+0x233/0x310
 [<ffffffff81121645>] slow_work_thread+0x205/0x360
 [<ffffffff810a6ac0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81121440>] ? slow_work_thread+0x0/0x360
 [<ffffffff810a662e>] kthread+0x9e/0xc0
 [<ffffffff8100c28a>] child_rip+0xa/0x20
 [<ffffffff810a6590>] ? kthread+0x0/0xc0
 [<ffffffff8100c280>] ? child_rip+0x0/0x20
Code:  Bad RIP value.
RIP  [<0000000000000002>] 0x2
 RSP <ffff880100003dd8>
CR2: 0000000000000002

Does anyone understand what caused it to crash? If there's any programs to help diagnose the issue, I'm open to hearing them.

4
+50

Quick Answer:

This looks related to kernel bug #13998 (see below for how I came to this conclusion), which has not been reproduced in later kernel versions. If this is the case, upgrading to a newer kernel (or a newer version of CentOS, same difference), should resolve the issue which is related to the fscache module.

Clues to fscache being the problem:

PANIC: "BUG: unable to handle kernel NULL pointer dereference at 000000000000002"

Means the kernel tried to load a memory address that made no sense.

COMMAND: "kslowd002"

This is the command the kernel was trying to run when it panicked. This does not necessarily mean this was the command that caused the crash, but it's a good place to start. What is kslowd? Read up on it here.

In the backtrace:

#9 [ffff880100003dd8] fscache_object_slow_work_execute at ffffffffa0460e9f [fscache]

Is the last procedure that runs before:

[exception RIP: unknown or invalid address]

This is the NULL pointer that the kernel couldn't dereference, in other words the place the kernel tried to look in memory but couldn't because it doesn't exist. This is a known bug with fscache that was, apparently, resolved in later kernel versions.

Here is a CentOS-6 specific bug report (#0007782) for the same issue that was not resolved. The recommendation from CentOS is also to make sure the kernel is the newest available version, which may in your case required an upgrade to the next stable major release of CentOS.

For more information on reading these crash dumps, I highly recommend this tutorial: http://www.dedoimedo.com/computers/crash-analyze.html

4
  • Hello, thank you for your input. I am running kernel release 2.6.32-642.1.1.el6.x86_64 which is the latest version available for CentOS 6.8 so I'm unable to update the kernel furthermore. Since it's already the latest version, I don't think it's the issue unless you believe updating to CentOS 7 will fix it. I'll read up on the crashdump link to see if I can pull any information from my server's dump file. Thanks again. – PolkaRon Aug 16 '16 at 16:12
  • The bug was reported on kernel 2.6.38.8. CentOS 7 uses the 3.x series of kernels, so upgrading to 7 would give you a newer kernel version (for comparison, the latest kernel release "in the wild" is 4.x). I see that CentOS was releasing patches for your kernel as recently as May of this year, so it's odd a patchfix for this issue hasn't been pushed down given this bug was reported in 2009. Short of waiting for a response to your bug report, unless there are any compelling reasons not to do so I would recommend an upgrade as generally advisable anyway. – Josh Benson Aug 16 '16 at 16:37
  • I edited my answer to provide a link to a CentOS bug report for this issue that was not resolved (because the original reporter stopped responding), but where a kernel/system upgrade was recommended to resolve it. – Josh Benson Aug 16 '16 at 20:37
  • Got it. I will be saving an upgrade to CentOS7 as a last resort. – PolkaRon Aug 16 '16 at 21:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.