I have a a problem with uefi grub2 booting, which means that the system cannot even reach grub console, let alone linux.

Instead it cycles though the boot process.

While I managed to 'fix' the problem by copying the uefi partition from a similar os, and modifying the grub.cfg, I'm left wondering how to debug this kind of problem.

I have seen no information about debugging grub, when you can't even reach the console, nor can I find any information about what goes into any of the files except the grub.cfg file.

I have seen lots of information describing what files are present, but nothing on whether the file's contents vary between grub installs.

Should I even consider this a grub debugging issue? Is it more of a uefi debugging issue?

To be clear, this question is about diagnostic methodologies for analyzing the problem, as opposed attempting a series of candidate solutions. For which there is a wealth of information on this site.

  • Could you tell more about the boot process cycle? Does it show anything? It could be UEFI, BIOS, Grub ... By the way, are you asking about a problem you cannot longer reproduce? (since you mentioned the fix) – Eduardo Trápani May 29 '20 at 1:15
  • I can reproduce every time, I have an image I have on USB drive, which on multiple machines will cause reboot cycles, I have another that doesn't. While I have modified the problematic usb drive, I have a backup of the EFI partiton contents, and also a full image of the drive. I'm trying to focus this question on debug methodologies, as there seems to be a lot of suggestions as to what to try, and do already have a work around. I want to be prepared for next time I have a grub issue. – rob May 29 '20 at 11:31
  • Regarding the boot process cycle, it starts with BIOS splash page, then some text blinks up, then back to BIOS splash page, rinse-repeat. Tricks to pause the process after the bios splash page would be greatly appreciated. – rob May 29 '20 at 11:56
  • One avenue for debugging would be to look at efi variables. For instance ' efibootmgr -v' or sudo cat /sys/firmware/efi/efivars/Boot0002-8be4df61-93ca-11d2-aa0d-00e098032b8c. If the efi images would write to efi vars, then maybe could read them on next successful boot? – rob May 29 '20 at 13:46
  • And here is someone who was debugging a *.efi image while booting... bugs.launchpad.net/ubuntu/+source/shim/+bug/1087501 – rob May 29 '20 at 14:14

This question asks about a whole class of problem where you can't boot your system in order to fix it. The question of how to fix a system that won't even get to a console is always the same: Use another system.

This might mean plugging your hard disk into another machine, but I don't recommend that. Firstly it might mean voiding your warranty with a screwdriver but secondly it might not work; it might hide the root cause of the problem.

Your alternative then is to boot a different OS on the same hardware. This can be done with a Live CD or Live USB.

Which distribution you use doesn't really matter but the actual OS (Kernel) and architecture must be at least from the same family. Eg: don't try to fix a BSD system with a Linux one, or an x86_64 system with an x86 one.

It doesn't usually matter which distribution you use though. Eg: of you are trying to fix a Centos box and all you have is an Ubuntu USB then that's fine.

For Linux Ubuntu is actually a good candidate because you can just use it's install disk and select the option "Try Ubuntu". That comes with enough drivers for most machines. But other options are available.

Steps to get a recovery console through Linux:

  1. Boot your Live disk
  2. Switch to root if not already done (sudo -i, or su -)
  3. Locate and mount your root / partition partition to /mnt. The blkid command can help you figure out what to mount. Eg:

    /dev/sda7: LABEL="Ubuntu_Boot" UUID="737d31ca-d399-4baa-a9e4-ec907b95f5ca" TYPE="ext4" PARTUUID="c3af19e0-e7d2-4867-b750-2b723208c535"
    /dev/sda2: UUID="F6C4-4D7F" TYPE="vfat" PARTLABEL="EFI system partition" PARTUUID="8663827c-80b4-4653-9df8-7ff56eebe6e2"
    /dev/sda6: LABEL="Ubuntu_Boot" UUID="3f91876b-7b58-428d-9249-756a62f8f839" TYPE="ext4" PARTLABEL="Ubuntu_Boot" PARTUUID="2ae7eab6-e068-4bc8-a4a3-fbd386fc8099"
    mount /dev/sda7 /mnt
  4. Mount any other partitions for your system. Remember you can now read /mnt/etc/fstab to remind you what these are. And if you are using UEFI you must mount your EFI partition. Eg:

    mount /dev/sda6 /mnt/boot
    mount /dev/sda2 /mnt/boot/efi
  5. Patch you mounted system with kernel provided shares:

    mount --bind /proc /mnt/proc
    mount --bind /sys /mnt/sys
    mount --bind /dev /mnt/dev
    mount --bind /tmp /mnt/tmp
  6. chroot into your new environment:

    chroot /mnt /bin/bash

This process gets you to a working console based entirely on your installed system. So commands like update-grub and grub-install should work as normal. And you can of course edit /etc/grub.d as well.

  • Ok, I don't think this quite answers the question of how to debug the issue. But does provide a strategy to attempt to fix the issue. I used this kind of strategy, but was unsuccessful. I will probably ask another question regarding how grub builds the contents of an EFI partition.... if it's not already answered. – rob May 29 '20 at 20:07
  • @rob I'm not quite certain I follow your difference between debugging and fixing then. Had you intended to specifically mean diagnose but not fix? Perhaps an edit to your question might make this a little clearer. – Philip Couling May 30 '20 at 10:06
  • Yes, that's a good idea. I think I'll add the diagnostic rider to the question, as that should clarify the question without getting into the philosophy of it. But regarding the philosophy, I consider debugging to be a cycle of observation, diagnostics, attempts at fixes, or other interventions, repeating till get close enough to root cause to solve problem – rob May 30 '20 at 12:57
  • @rob yeah. I see your point. I think many see it more literally as the act of removing bugs. – Philip Couling May 30 '20 at 13:42

Different Linux distributions use GRUB2 with UEFI in different ways: Debian/Ubuntu seems to set up a mini grub.cfg within the EFI System Partition (ESP for short) that just points GRUB2 to the actual configuration file in /boot i.e. on a different filesystem. RedHat seems to put the actual GRUB configuration into ESP.

The contents of the actual grubx64.efi file can also vary between distributions, because it is produced using grub-mkimage. It includes:

  • the GRUB core image
  • a number of GRUB modules: at minimum, the modules needed to access the filesystem where other GRUB modules are stored (typically /boot/grub/x86_64-efi on Debian), but on UEFI systems, it is possible to include all the modules into the grubx64.efi file.
  • information on the initial GRUB root filesystem and prefix (i.e. where GRUB looks for the GRUB module directory and configuration file). On UEFI GRUB, this is typically /EFI/<name of distribution> to refer to the directory on the ESP GRUB is installed into.
  • optionally, an embedded GRUB configuration file
  • optionally, a memdisk image file
  • optionally, a public key/certificate file for checking signatures on GRUB modules and other files (for implementing a Secure Boot-like check on systems with no Secure Boot functionality available)

The modularity of GRUB is the result of its origin on PCs with MBR-style boot, which required the GRUB to fit in the unused space between the MBR and the beginning of the first partition. Depending on partitioning, that space can be quite small. In UEFI the size limitations are far less strict, but the module mechanism is still available if needed.

Secure Boot will prevent GRUB from loading executable code from files that are not properly signed and using the Windows PE32 binary format. GRUB modules use Unix-style ELF binary format instead, so when Secure Boot is in use, all the needed modules need to be packed into the main grubx64.efi file so that there will be no need to load separate executable modules. The Linux kernel can have a built-in EFI stub that makes it effectively a PE32 binary file, so it won't have this problem when properly signed.

The Secure Boot shim bootloader shimx64.efi is signed by Microsoft, so it will be acceptable by default by basically all Secure Boot implementations, and on load, it will add one or two more public keys to the Secure Boot allowed list: the distribution's key, and optionally the MOK key generated for the system owner. This will allow the use of Linux and customized kernels, even if the system vendor will not include Linux distributions' public keys to their Secure Boot implementation and the firmware won't allow the user to edit the firmware-based list of allowed Secure Boot keys.

(The shim must not accept entirely unsigned binaries for execution, for then it would become a "Secure Boot circumvention device" and Microsoft and any other Secure Boot signers should refuse to sign it.)

Some Secure Boot systems will stop the boot process and display a warning message if you're trying to use a bootloader that has no valid Secure Boot signature. Others will just ignore the invalid bootloader and proceed to the next boot option. If your system has Secure Boot enabled and has no other valid boot sources, this might be the cause of the cycling you're seeing: you might be trying to load a version of GRUB that is intended for UEFI with Secure Boot disabled.

Without knowing the name and version of the Linux distribution you're using, and the make and model of your system or motherboard, it will be hard to give any more detailed answers.

  • 1
    Thanks, interesting foundational notes, although doesn't really help to debug the system. I'm not focusing on this distro, as .. 1. I'm looking for general principals for debugging grub installs. 2. I'm probably about to change from Ubuntu to other distro, so any advice that is currently relevant will probably become obsolete. – rob May 29 '20 at 11:40
  • I've had a look at the boot partition and the UEFI specification, now I always thought that the bios would execute the image in /EFI/BOOT/BOOTX64.EFI but it turns out that this is only true for removable storage. As I'm using USB drives for test I think this is true. For boot from internal drive, I think your answer is correct, in fact looking in BOOT0001 variable in efivars, it appears to point to shimx64.efi – rob May 29 '20 at 12:47
  • After some more reading, it seems that the external storage method is a panic method for selecting boot image if the internal storage doesn't have a path, and will typically end up with a dance resulting in a UEFI var being generated for the internal storage device. As described here: blog.uncooperative.org/blog/2014/02/06/the-efi-system-partition – rob May 29 '20 at 13:41
  • That "dance" is very OS-dependent: Windows may call it "self-healing", and according to that link Fedora is trying something similar. I don't recall seeing other/enterprise distributions doing anything that complex with the removable/fallback boot path quite yet, but that may be changing. But yes, checking the firmware boot variables is an entirely new part of boot troubleshooting when moving from MBR to UEFI, and sadly the interface provided by the system firmware (UEFI "BIOS Setup") for that tends to vary a lot between HW vendors. – telcoM May 29 '20 at 14:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.