Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm running Ubuntu 12.04 on AWS/EC2 and having a large number of hosts going belly up. I'm trying to enable kernel dumping, but when I simulate a kernel panic, there's no .crash file written anywhere on the file system.

I followed the instructions here: https://wiki.ubuntu.com/Kernel/CrashdumpRecipe

And things seem set up correctly:

# cat /proc/cmdline 
root=LABEL=cloudimg-rootfs ro console=hvc0  crashkernel=384M-2G:64M,2G-:128M

# dmesg |grep crash
[    0.000000] Command line: root=LABEL=cloudimg-rootfs ro console=hvc0  crashkernel=384M-2G:64M,2G-:128M
[    0.000000] Reserving 64MB of memory at 832MB for crashkernel (System RAM: 1708MB)
[    0.000000] Kernel command line: root=LABEL=cloudimg-rootfs ro console=hvc0  crashkernel=384M-2G:64M,2G-:128M

# cat /sys/kernel/kexec_crash_loaded
1

But when I execute:

# echo c | sudo tee /proc/sysrq-trigger

The system reboots as expected, but no 'crash' file of any kind is generated. What might I be doing wrong?

share|improve this question
    
Anything of note in /var/log/messages? –  Banjer Jan 8 '13 at 23:52
    
Nothing unusual in /var/log/syslog, kern.log, nor dmesg unfortunately. –  Stephan Jan 8 '13 at 23:54

1 Answer 1

Make sure the kdump initscript is enabled. kexec_crash packages rely upon an initscript to bypass the normal startup routine. It determines whether or not the current invocation of init was one that was invoked by a crash, and uses that to determine whether or not the previous running state needs to be dumped before performing a true reboot.

That said, if your testing system isn't small enough to fit into 64Mb without you noticing that every other crash is reducing your total memory, this is probably not what is going on.

The main thing you need to look for is whether the second init is firing. Immediately after you crash the system, you should see initscript startup sequences on the console that are not preceded by a reboot.

  • If this is not happening, your crash kernel is not firing at all.
  • If this is happening and you get dropped to a prompt, your initscript is not doing its job. (either it is not enabled or not detecting the post-crash state)
  • If this is happening, the second init fires, the system reboots, init starts again, and despite all of this you still have no file...you need to troubleshoot what is going on right before the kdump initscript issues the reboot. Ironically enough, one of the better methods is to disable the initscript and run the commands by hand. (caution: make sure your services can fit into the memory of the crash kernel before attempting this!)
share|improve this answer
    
Thank you so much for the suggestions! I'll dig into it now. As background, we're investigating AWS EC2 instances simply falling over at a rate we've never had before, and Amazon claims there's been nothing at all reported wrong with the underlying hardware; thus trying to rule out kernel panics, etc. –  Stephan Feb 2 '13 at 1:38
    
@Stephan Any luck? The question is still open. –  Andrew B Jul 10 '13 at 4:52

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.