After some recent updates, my computer no longer boots! Here's what I could determine:
- This is a very recent computer that was provided to me by corporate IT. It has a recent Intel CPU (Skylake generation).
- The computer runs Ubuntu 16.04.
- The computer last booted correctly some time in March. The problem is presumably due to a software update or a hardware bug.
- I have another computer running 16.04 with pretty much the same software installed (I used
apt-clone), and it works just fine. It has different hardware (also amd64, but different CPU, different GPU, etc.).
- The kernel does start, the initrd works correctly. When I boot with a splash screen in graphics mode, I get prompted for the password for my dm-crypt volume, and the last thing I see is that it's mounted successfully.
- The hang occurs before I get a login prompt. When the computer hangs, it's a hard hang. Even Alt+SysRq doesn't respond. The CPU is evidently pegged at 100% since the fans turn on at full blast.
- I still have the kernel I was running before rebooting. When I select this kernel in the Grub menu, I get the same lockup. So it looks like this is a pre-existing kernel bug which gets triggered by something else — but what?
- If I switch off the splash screen (remove
linuxcommand line in Grub), I see a number of services starting, then it locks up.
I can get a root shell by adding
linuxcommand line in Grub. I can even get further by adding
This starts a number of services and runs a root shell on tty9.
- If I run
systemctl start multi-user.targetfrom that root shell, the computer locks up. So presumably the problem is triggered by one of these services.
- I ran
systemctl list-dependencies multi-user.targetto see what services get started. I manually started the listed dependencies one by one, and everything started just fine.
So this looks like a hardware bug (since it occurs on one computer but not on the other one) that gets triggered by some software. But what software? Since the computer locks up so hard, I can't get any logs. I can't even get any useful console output.
Useful debugging techniques:
- Alt+SysRq: magic SysRq key, which lets you do things such as an emergency reboot. It accesses the kernel at a very low level, so it works in all but the worst crashes. In my case, Alt+SysRq doesn't respond, which shows how deep the crash goes.
- To modify the boot parameters, press and hold Shift a few seconds after switching the power on. You need to press it after the BIOS has initialized the keyboard, but before the operating system boots. This makes the Grub menu appear.
- At the Grub menu, press e to edit the command line for a menu entry. To change the Linux boot parameters, navigate to the line that starts with
linux. On a modern Ubuntu, you'll find old kernels under “Advanced options for Ubuntu”. Once you've made the desired changes to the command line, press Ctrl+x to boot. Any change you make here are for this boot only, they aren't saved to disk.
- Some useful options on the
quiet nosplashhides almost all boot messages. Remove them to get messages on the console during boot, which is necessary to have any chance of diagnosing problems.
recoverygives you a root shell with almost no services. You'll need to know the root password. The “recovery mode” menu entry uses this.
init=/bin/shgives you a root shell with no services at all. To resume normal boot, run
exec init. You can pass systemd options at this point, e.g.
exec init --unit=basic.targetto start init and a few services (note that this does not start any way to log in, so you'd better have a shell running on another console). Note that the root filesystem is mounted read-only; run
mount -o remount,rw /to be able to write to it.
systemd.unit=basic.targetstarts a very basic set of services. Note that this does not include any way to log in! You can make this the default by running
systemctl set-default basic.targetat a root prompt. To restore the original default target, run
systemctl set-default graphical.target(or
systemctl set-default multi-user.targetfor a server with no GUI).
systemd.debug-shellstarts a root shell on tty9. You can enable this for every boot by running
systemctl enable debug-shellat a root prompt. Don't forget to disable this after you've solved the problem with
systemctl disable debug-shell. Press Alt+F9 to switch to tty9.
- See also Fedora systemd tips, Arch Linux boot problem tips.