Trouble Logging Kernel Panic for Debugging

Question

I'm running Ubuntu 12.04 on AWS/EC2 and having a large number of hosts going belly up. I'm trying to enable kernel dumping, but when I simulate a kernel panic, there's no .crash file written anywhere on the file system.

I followed the instructions here: https://wiki.ubuntu.com/Kernel/CrashdumpRecipe

And things seem set up correctly:

# cat /proc/cmdline 
root=LABEL=cloudimg-rootfs ro console=hvc0  crashkernel=384M-2G:64M,2G-:128M

# dmesg |grep crash
[    0.000000] Command line: root=LABEL=cloudimg-rootfs ro console=hvc0  crashkernel=384M-2G:64M,2G-:128M
[    0.000000] Reserving 64MB of memory at 832MB for crashkernel (System RAM: 1708MB)
[    0.000000] Kernel command line: root=LABEL=cloudimg-rootfs ro console=hvc0  crashkernel=384M-2G:64M,2G-:128M

# cat /sys/kernel/kexec_crash_loaded
1

But when I execute:

# echo c | sudo tee /proc/sysrq-trigger

The system reboots as expected, but no 'crash' file of any kind is generated. What might I be doing wrong?

Answer 1

Make sure the kdump initscript is enabled. kexec_crash packages rely upon an initscript to bypass the normal startup routine. It determines whether or not the current invocation of init was one that was invoked by a crash, and uses that to determine whether or not the previous running state needs to be dumped before performing a true reboot.

That said, if your testing system isn't small enough to fit into 64Mb without you noticing that every other crash is reducing your total memory, this is probably not what is going on.

The main thing you need to look for is whether the second init is firing. Immediately after you crash the system, you should see initscript startup sequences on the console that are not preceded by a reboot.

• If this is not happening, your crash kernel is not firing at all.
• If this is happening and you get dropped to a prompt, your initscript is not doing its job. (either it is not enabled or not detecting the post-crash state)
• If this is happening, the second init fires, the system reboots, init starts again, and despite all of this you still have no file...you need to troubleshoot what is going on right before the kdump initscript issues the reboot. Ironically enough, one of the better methods is to disable the initscript and run the commands by hand. (caution: make sure your services can fit into the memory of the crash kernel before attempting this!)