wrong grub prefix (efi) `grub-install` and `update-grub` doesn't fix it

Question

problem:

When booting, grub goes to the prompt (grub>, not grub rescue>). I need to type in:

set prefix=(hd0,gpt8)/boot/grub
insmod normal
normal

To get the normal grub menu.

Every post I found so far was solved by doing grub-install and/or update-grub once you manage to boot your system.
I tried these but the problem persists.

details:

I dual boot with windows 10 and pop-os (ubuntu-based). I have an acer laptop with boot mode set to UEFI and secure boot enabled.

My partitions look like this:

$ lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT,UUID,LABEL,PARTUUID,PARTTYPE
NAME            SIZE FSTYPE MOUNTPOINT UUID                                 LABEL    PARTUUID                             PARTTYPE
sda           238.5G                                                                                                      
├─sda1          529M ntfs              6272EE1672EDEF2B                     Recovery b593e2b8-992e-4d79-9074-d990ba21d10c de94bba4-06d1-4d40-a16a-bfd50179d6ac
├─sda2          100M vfat              64EE-A907                                     6ef6e12c-3858-4a83-a1dd-8297719bd477 c12a7328-f81f-11d2-ba4b-00a0c93ec93b
├─sda3           16M                                                                 1b6ba39a-e528-4cf9-b158-cd1a9312afc2 e3c9e316-0b5c-4db8-817d-f92df00215ae
├─sda4        117.2G ntfs              8E6008B56008A655                              c2f2d5bf-16a3-4e99-a6db-2385b36f3f46 ebd0a0a2-b9e5-4433-87c0-68b6b72699c7
├─sda5         58.6G ntfs              F6D0CCD7D0CC9EED                     Storage  c10e8d39-26c3-4d3b-8548-0aa97816ba0b ebd0a0a2-b9e5-4433-87c0-68b6b72699c7
├─sda6          477M swap              74da5edc-3b6c-4644-a151-6b93562c8fa4          408e91bd-c7e7-4ab8-a2ac-bb2bb7fde375 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f
│ └─cryptswap 476.5M swap   [SWAP]     2d2336c7-64c1-4d53-8e99-f66f9b93bef5                                               
├─sda7          600M vfat   /boot/efi  7565-8082                                     a8c1ff00-3790-4ca0-a360-642e6f1859f0 c12a7328-f81f-11d2-ba4b-00a0c93ec93b
└─sda8           61G ext4   /          d40fe3bd-0749-4c29-9e9b-97a064a659dd          8a3b8567-c511-48cd-a3fa-776b556d17da 0fc63daf-8483-4772-8e79-3d69d8477de4

• Partitions 1-4 were created by windows, the bootloader is on sda2.
• Partition 5 is created by me to easily share files between windows/linux.
• The remaining partitions were created by pop-os (after completing the windows installation), the pop-os bootloader is on sda7.

I installed grub as:

$ sudo apt install grub-efi
$ sudo grub-install --bootloader-id=grub
Installing for x86_64-efi platform.
Installation finished. No error reported.
$ sudo update-grub
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.4.0-7642-generic
Found initrd image: /boot/initrd.img-5.4.0-7642-generic
Found linux image: /boot/vmlinuz-5.4.0-7634-generic
Found initrd image: /boot/initrd.img-5.4.0-7634-generic
Found linux image: /boot/vmlinuz-5.4.0-7629-generic
Found initrd image: /boot/initrd.img-5.4.0-7629-generic
Found linux image: /boot/vmlinuz-5.4.0-7626-generic
Found initrd image: /boot/initrd.img-5.4.0-7626-generic
Found Windows Boot Manager on /dev/sda2@/efi/Microsoft/Boot/bootmgfw.efi
Adding boot menu entry for UEFI Firmware Settings
done

After which my /boot directory looks like this:

$ sudo tree -a /boot
/boot
├── config-5.4.0-7626-generic
├── config-5.4.0-7629-generic
├── config-5.4.0-7634-generic
├── config-5.4.0-7642-generic
├── efi
│   ├── c0cc91f7cdfcb9a597d9db525eb08842
│   ├── EFI
│   │   ├── BOOT
│   │   │   └── BOOTX64.EFI
│   │   ├── Linux
│   │   ├── grub
│   │   │   ├── grub.cfg
│   │   │   └── grubx64.efi
│   │   ├── Pop_OS-d40fe3bd-0749-4c29-9e9b-97a064a659dd
│   │   │   ├── cmdline
│   │   │   ├── initrd.img
│   │   │   ├── initrd.img-previous
│   │   │   ├── vmlinuz.efi
│   │   │   └── vmlinuz-previous.efi
│   │   └── systemd
│   │       └── systemd-bootx64.efi
│   ├── loader
│   │   ├── entries
│   │   │   ├── Pop_OS-current.conf
│   │   │   └── Pop_OS-oldkern.conf
│   │   ├── loader.conf
│   │   └── random-seed
│   └── System Volume Information
│       ├── AadRecoveryPasswordDelete
│       └── ClientRecoveryPasswordRotation
├── grub
│   ├── fonts
│   │   └── unicode.pf2
│   ├── grub.cfg
│   ├── grubenv
│   └── x86_64-efi
│       ├── acpi.mod
|       < files omitted ... > 
│       └── zstd.mod
├── initrd.img -> initrd.img-5.4.0-7642-generic
├── initrd.img-5.3.0-7642-generic
< files omitted ... > 
├── initrd.img.old -> initrd.img-5.4.0-7634-generic
├── System.map-5.4.0-7626-generic
< files omitted ... > 
├── vmlinuz -> vmlinuz-5.4.0-7642-generic
├── vmlinuz-5.4.0-7626-generic
< files omitted ... > 
└── vmlinuz.old -> vmlinuz-5.4.0-7634-generic

16 directories, 314 files

The files under /boot/efi/EFI/grub, /boot/efi/EFI/BOOT and /boot/grub are newly added by grub-install.

In addition, this looks fine to me:

$ cat /boot/efi/EFI/grub/grub.cfg 
search.fs_uuid d40fe3bd-0749-4c29-9e9b-97a064a659dd root hd0,gpt8 
set prefix=($root)'/boot/grub'
configfile $prefix/grub.cfg

After rebooting, i hit F2 and managed to add EFI/grub/grubx64 to the top of the boot order.

After saving & exiting I land into the grub prompt.

Then, using the set command I see the following output (some entries omitted):

grub> set
cmdpath=(hd0,gpt7)/EFI/grub
grub_cpu=x86_64
grub_platform=efi
prefix=(hd0,gpt7)/EFI/ubuntu
root=(hd0,gpt7)

I'm lost now, it seems that the correct bootloader was loaded (cmdpath=(hd0,gpt7)/EFI/pop), but the settings in EFI/grub/grub.cfg are completely ignored. Where is it getting these incorrect settings from?

After fixing prefix as decribed above, I get a fully functional grub menu with an entry for 1. pop-os, 2. pop-os fallback images, 3. windows and 4. the boot setup. I tried all options and they all work.

So I managed to boot into popOs, efibootmgr -v tells me this:

$ efibootmgr -v
BootCurrent: 0003
Timeout: 0 seconds
BootOrder: 0003,0001,0002,0000,2001,2002,2003
Boot0000* Windows Boot Manager     HD(2,GPT,6ef6e12c-3858-4a83-a1dd-8297719bd477,0x109000,0x32000)/File(\EFI\Microsoft\Boot\bootmgfw.efi)RC
Boot0001* systemd                  PciRoot(0x0)/Pci(0x17,0x0)/Sata(2,0,0)/HD(7,GPT,a8c1ff00-3790-4ca0-a360-642e6f1859f0,0x160d3000,0x12c000)/File(\EFI\systemd\systemd-bootx64.efi)A01 ..
Boot0002* HDD0: HFS256G39TND-N210A PciRoot(0x0)/Pci(0x17,0x0)/Sata(2,0,0)/HD(7,GPT,a8c1ff00-3790-4ca0-a360-642e6f1859f0,0x160d3000,0x12c000)RC
Boot0003* grub                     PciRoot(0x0)/Pci(0x17,0x0)/Sata(2,0,0)/HD(7,GPT,a8c1ff00-3790-4ca0-a360-642e6f1859f0,0x160d3000,0x12c000)/File(\EFI\grub\grubx64.efi)A01 ..
Boot0004* EFI USB Device           RC
Boot0005* EFI DVD/CDROM            RC
Boot0006* EFI Network              RC
Boot2001* EFI USB Device           RC
Boot2002* EFI DVD/CDROM            RC
Boot2003* EFI Network              RC

what I expect should happen

1. I start grub, at (hd0,gpt7)/EFI/grub/grub.efi from the UEFI firmware.
2. grub reads the config file at (hd0,gpt7)/EFI/grub/grub.cfg. This sets root to (hd0,gpt8) and prefix to (hd0,gpt8)/boot/grub.
3. grub loads the configuration at (hd0,gpt8)/boot/grub/grub.cfg.
4. grub presents me with a menu, I choose the OS I want to load and go on with my day.

what actually happens

1. I start grub, at (hd0,gpt7)/EFI/grub/grub.efi from the UEFI firmware.
2. grub does not read the config file at (hd0,gpt7)/EFI/grub/grub.cfg. Instead it sets root to (hd0,gpt7) and prefix to (hd0,gpt7)/boot/ubuntu.
   
   I manually fix the incorrect prefix as described above.
3. grub loads the configuration at (hd0,gpt8)/boot/grub/grub.cfg.
4. grub presents me with a menu, I choose the OS I want to load and go on with my day.

things I tried:

• disabling secure boot (no difference)
• re-running grub-install and upgrade-grub (no difference)
• sudo grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=grub --recheck --debug /dev/sda &> grub.log (no difference, output here)

Answer 1

When grubx64.efi is generated by grub-mkimage, a default prefix value can be embedded into it.

In Debian, this is set to /EFI/debian when grub-mkimage is called by grub-install; in Ubuntu, the prefix will be /EFI/ubuntu respectively. This causes GRUB to first read a mini-configuration file located as /EFI/(debian|ubuntu)/grub.cfg, which typically contains only a few lines to redirect GRUB into wherever the /boot directory/filesystem is located, to read the real configuration file from there:

<any configuration lines required to cover complications like software RAID, disk encryption or LVM>
search.fs_uuid <UUID of the filesystem that contains /boot>
set prefix=($root)'/boot/grub' # or just ($root)/grub if /boot is a separate filesystem 
configfile $prefix/grub.cfg

If you are using Pop!OS's GRUB package, it might have a similar left-over "ubuntuism" in its grub-install/grub-mkimage defaults, if Pop's packagers have not paid enough attention. And since Pop's default UEFI bootloader seems to be gummiboot rather than GRUB, such a bug might have gone unnoticed.

If you have installed the grub-efi-amd64-signed package for Secure Boot compatibility, your original /boot/efi/EFI/pop/grubx64.efi may in fact be different from /boot/grub/x86_64-efi/grub.efi, as the former has gone through the Secure Boot signing process and usually has essentially all the GRUB modules built-in.

Perhaps the Pop!OS developers have used Ubuntu's grub-efi-amd64-signed package as-is, without applying the changes they've made to the non-SecureBoot-signed version?

---

Background:

When grub-install is used with the --uefi-secure-boot option, or if Secure Boot is detected, then instead of using grub-mkimage to build an optimized grubx64.efi for the system GRUB is being installed to, it uses a pre-compiled and Secure Boot-signed /usr/lib/grub/x86_64-efi-signed/grubx64.efi.signed.

At least in Debian, this is built using reproducible build techniques, so with the corresponding version of the source package and a detached Secure Boot signature file, anyone can compile it for themselves and verify that the result will be absolutely identical to the pre-built version. Getting that part working right was apparently the major reason why it took a while to get Secure Boot support into Debian.

Answer 2

The exact reason for this problem is not yet entirely clear, but it may have something to do with incorrectly configured grub packages by the popOs maintainers.

I managed to work around it by copying the grub files created by grub-install to /boot/efi/EFI/ubuntu. If you're on popOs and did grub-install without any options, then the following should work:

sudo mv /boot/efi/EFI/pop /boot/efi/EFI/ubuntu

An alternative workaround, which I haven't tested is:

sudo cp /boot/grub/x86_64-efi/grub.efi /boot/efi/EFI/pop/grubx64.efi

For more info, see the bug report.

Answer 3

Quoted from Sam De Meyer Answer,

An alternative workaround, which I haven't tested is:

sudo cp /boot/grub/x86_64-efi/grub.efi /boot/efi/EFI/pop/grubx64.efi

Moving grub.efi from /boot/grub/x86_64-efi to the efi partition (which one on my os is on /boot/efi/EFI/neon/grubx64.efi) makes the GRUB's theme could not be used even though I have applied it in /etc/default/grub with GRUB_THEME configuration. The default blue grub theme is used instead.

The solution for me to fix this issue with keeping my theme works is just to copy the /boot/efi/EFI/neon/grub.cfg to /boot/efi/EFI/ubuntu/grub.cfg

sudo cp /boot/efi/EFI/neon/grub.cfg /boot/efi/EFI/ubuntu/grub.cfg

Before doing that, you should run sudo grub-install first.