3

I'm on Linux 4.2.0, the Ubuntu Wily HWE kernel for Ubuntu 14.04 (which is what I'm running).

There's a nasty bug on MacBook 11,4 and 11,5 models where the laptops won't actually ever shut off, they seem to reach the Power down kernel message and then just hang without ever shutting off. This bug is likely preventing suspend and resume as well. Interestingly enough, it doesn't manifest when reboot is called, only when trying to just halt the machine.

The details aren't so relevant, but what is relevant is how I could debug this. Is there a way to strace or debug the kernel while it's shutting down and observe the output somehow? I'm fairly certain that the power off logic is making a syscall that is just hanging forever and not returning for some strange reason. If I can figure out which syscall isn't working, I could then continue onward to figure out why it isn't working and what specifically about the hardware is causing the shutdown to fail.

Is there a standard practice for debugging a kernel shut down? Do I need special hardware? I'm fine writing patches for the kernel, but I don't even know where to begin looking for this issue unless I can find the syscall that isn't working.

  • Was there anything additional you needed to the comment below? – Jeff Coleman Mar 1 '16 at 13:43
  • check your system log message, this might already tell you which syscall is not responding. – Winston Mar 2 '16 at 23:15
5
+50

A rough outline on what you may need to do:

  1. Download source and compile the kernel to enable debugging (config -> kernel hacking)
  2. Install kgdb patches
  3. Connect to target via a serial port from another machine. The target could be a different machine or could be an emulator like qemu or bochs.
  4. From the source code, identify the routine for shutdown
  5. Set a break point from gdb and step through until you see the hang.

You can find more info on these steps by searching for kernel debugging using kgdb.

This guide can help you set up your debugging environment.

2

Check the kernel log:

nano /var/log/kern.log

It will have all of the relevant information if the shutdown was done properly and should give you the debugging information you need.

You can also check dmesg

nano /var/log/dmesg

You will also see backups of these from previous startups/shutdowns the will be appended like this

dmesg.0 kern.log.1

And there are archives going back further there as well.

You can make these more verbose by adding further options in your kernel:

  • CONFIG_PRINTK_TIME - add time stamps to dmesg

  • CONFIG_DEBUG_KERNEL - turn on kernel debugging

  • CONFIG_DETECT_HUNG_TASK - good for figuring out what's causing a
    kernel freeze

  • CONFIG_DEBUG_INFO - ensures you can decode kernel

  • CONFIG_EARLY_PRINTK

  • CONFIG_LOG_BUF_SHIFT=21 - sets the kernel buffer log size to the
    biggest buffer

  • CONFIG_NETCONSOLE=m - compiles netconsole as a module

  • What I'd really like to be able to see is which syscalls aren't returning during shutdown. I'm not sure that syslog or dmesg will contain this information. – Naftuli Kay Mar 1 '16 at 18:41
  • I'm fairly certain that in order to get something that low level you're going to have to patch the kernel code to send debug messages to dmesg – Jeff Coleman Mar 2 '16 at 15:30
1

I would take a guess that the hang is a device driver not returning from it's power state change call or similar - so there won't be any useful debug messages unless you're liberal with printk's.

To confirm this using a quick and dirty hack, blacklist all the non-essential drivers in your kernel bootup command line (as present in grub.cfg) - such as wi-fi, network, etc. then test by power cycling to identify any misbehaving driver code.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.