After some recent updates, my computer no longer boots! Here's what I could determine:
- This is a very recent computer that was provided to me by corporate IT. It has a recent Intel CPU (Skylake generation).
- The computer runs Ubuntu 16.04.
- The computer last booted correctly some time in March. The problem is presumably due to a software update or a hardware bug.
- I have another computer running 16.04 with pretty much the same software installed (I used
apt-clone
), and it works just fine. It has different hardware (also amd64, but different CPU, different GPU, etc.). - The kernel does start, the initrd works correctly. When I boot with a splash screen in graphics mode, I get prompted for the password for my dm-crypt volume, and the last thing I see is that it's mounted successfully.
- The hang occurs before I get a login prompt. When the computer hangs, it's a hard hang. Even Alt+SysRq doesn't respond. The CPU is evidently pegged at 100% since the fans turn on at full blast.
- I still have the kernel I was running before rebooting. When I select this kernel in the Grub menu, I get the same lockup. So it looks like this is a pre-existing kernel bug which gets triggered by something else — but what?
- If I switch off the splash screen (remove
splash
from thelinux
command line in Grub), I see a number of services starting, then it locks up. I can get a root shell by adding
init=/bin/sh
to thelinux
command line in Grub. I can even get further by addingsystemd.unit=basic.target systemd.shell
This starts a number of services and runs a root shell on tty9.
- If I run
systemctl start multi-user.target
from that root shell, the computer locks up. So presumably the problem is triggered by one of these services. - I ran
systemctl list-dependencies multi-user.target
to see what services get started. I manually started the listed dependencies one by one, and everything started just fine.
So this looks like a hardware bug (since it occurs on one computer but not on the other one) that gets triggered by some software. But what software? Since the computer locks up so hard, I can't get any logs. I can't even get any useful console output.
Useful debugging techniques:
- Alt+SysRq: magic SysRq key, which lets you do things such as an emergency reboot. It accesses the kernel at a very low level, so it works in all but the worst crashes. In my case, Alt+SysRq doesn't respond, which shows how deep the crash goes.
- To modify the boot parameters, press and hold Shift a few seconds after switching the power on. You need to press it after the BIOS has initialized the keyboard, but before the operating system boots. This makes the Grub menu appear.
- At the Grub menu, press e to edit the command line for a menu entry. To change the Linux boot parameters, navigate to the line that starts with
linux
. On a modern Ubuntu, you'll find old kernels under “Advanced options for Ubuntu”. Once you've made the desired changes to the command line, press Ctrl+x to boot. Any change you make here are for this boot only, they aren't saved to disk. - Some useful options on the
linux
command line:quiet nosplash
hides almost all boot messages. Remove them to get messages on the console during boot, which is necessary to have any chance of diagnosing problems.recovery
gives you a root shell with almost no services. You'll need to know the root password. The “recovery mode” menu entry uses this.init=/bin/sh
gives you a root shell with no services at all. To resume normal boot, runexec init
. You can pass systemd options at this point, e.g.exec init --unit=basic.target
to start init and a few services (note that this does not start any way to log in, so you'd better have a shell running on another console). Note that the root filesystem is mounted read-only; runmount -o remount,rw /
to be able to write to it.systemd.unit=basic.target
starts a very basic set of services. Note that this does not include any way to log in! You can make this the default by runningsystemctl set-default basic.target
at a root prompt. To restore the original default target, runsystemctl set-default graphical.target
(orsystemctl set-default multi-user.target
for a server with no GUI).systemd.debug-shell
starts a root shell on tty9. You can enable this for every boot by runningsystemctl enable debug-shell
at a root prompt. Don't forget to disable this after you've solved the problem withsystemctl disable debug-shell
. Press Alt+F9 to switch to tty9.- See also Fedora systemd tips, Arch Linux boot problem tips.