LVM on LUKS FDE: GRUB won't mount my root partition / during boot even though cryptomount and cryptdevice are given in grub.cfg

Question

I have a triple boot setup (3xLinux). All Linuces share /home and swap, and their / partitions are next to each other. All reside in an LVM on LUKS:

# lsblk
NAME                     MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
nvme0n1                  259:0    0 951.8G  0 disk  
|-nvme0n1p1              259:1    0   800M  0 part  /boot/efi
|-nvme0n1p2              259:2    0    32G  0 part  
|-nvme0n1p3              259:3    0   619M  0 part  
`-nvme0n1p4              259:4    0 706.5G  0 part  
  `-cryptolvm            254:0    0 706.5G  0 crypt 
    |-cryptolvm-swap     254:1    0    32G  0 lvm   [SWAP]
    |-cryptolvm-home     254:2    0 430.0G  0 lvm   /home
    |-cryptolvm-centos   254:3    0    41G  0 lvm   /mnt/centos
    |-cryptolvm-arch     254:4    0    41G  0 lvm   /
    `-cryptolvm-opensuse 254:5    0    41G  0 lvm   /mnt/opensuse

openSUSE manages the GRUB2 (with full encryption, i.e. disk unlock password needed before grub menu which resides on the openSUSE system partition /).

Problem: Arch does not unlock the cryptodisk /dev/nvme0n1p4 and therefore cannot access its / during boot. It drops me to an emergency prompt.

This is the setup an Arch Linux:

mkinitcpio configuration (and yes, I have re-created /boot/initramfs-linux.img after changing this):

# grep crypt /etc/mkinitcpio.conf | tail -1
HOOKS=(base udev autodetect modconf keyboard block encrypt lvm2 filesystems fsck)

GRUB2 configuration on openSUSE:

# grep --after=18 Arch /mnt/opensuse/boot/grub2/grub.cfg
menuentry 'Arch Linux (rolling) (on /dev/mapper/cryptolvm-arch)' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'osprober-gnulinux-simple-aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa' {
    insmod part_gpt
    insmod cryptodisk
    insmod luks
    insmod gcry_rijndael
    insmod gcry_rijndael
    insmod gcry_sha256
    insmod lvm
    insmod ext2
    cryptomount -u 99999999999999999999999999999999
    set root='lvmid/VVVVVV-VVVV-VVVV-VVVV-VVVV-VVVV-VVVVVV/qqqqqq-qqqq-qqqq-qqqq-qqqq-qqqq-qqqqqq'
    if [ x$feature_platform_search_hint = xy ]; then
      search --no-floppy --fs-uuid --set=root --hint='lvmid/VVVVVV-VVVV-VVVV-VVVV-VVVV-VVVV-VVVVVV/qqqqqq-qqqq-qqqq-qqqq-qqqq-qqqq-qqqqqq'  aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
    else
      search --no-floppy --fs-uuid --set=root aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
    fi
    linuxefi /boot/vmlinuz-linux cryptdevice=UUID=99999999999999999999999999999999:cryptolvm root=/dev/mapper/cryptolvm-arch  resume=/dev/cryptolvm/swap splash=silent quiet showopts
    initrdefi /boot/initramfs-linux.img
}

LVM devices:

# vgs -v
  VG        Attr   Ext   #PV #LV #SN VSize   VFree    VG UUID                                VProfile
  cryptolvm wz--n- 4.00m   1   5   0 706.45g <120.51g VVVVVV-VVVV-VVVV-VVVV-VVVV-VVVV-VVVVVV

# lvs -v | grep arch
  arch     cryptolvm    1 -wi-ao----  <40.96g  -1  -1  254    4  qqqqqq-qqqq-qqqq-qqqq-qqqq-qqqq-qqqqqq

Disk/partition UUIDs:

# blkid | egrep '(p4|arch)'
/dev/nvme0n1p4: UUID="99999999-9999-9999-9999-999999999999" TYPE="crypto_LUKS" PARTUUID="cccccccc-cccc-cccc-cccc-cccccccccccc"
/dev/mapper/cryptolvm-arch: UUID="aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa" PARTUUID="cccccccc-cccc-cccc-cccc-cccccccccccc"

Workaround:

GRUB2 (or initramfs) drops me to a prompt because it cannot mount /dev/mapper/cryptolvm-arch to / (or, to /new_root). So during each boot I enter manually:

> cryptsetup open /dev/nvme0n1p4 cryptolvm
(...)
> mount /dev/mapper/cryptolvm-arch /new_root
> ^D

Why is this necessary? The mount is given twice (cryptomount and cryptdevice) in grub.cfg (and this is indeed used).

EDIT:

Perhaps this has to do with EFI? Would I receive an EFI error if it were? openSUSE boots via EFI, chain-loads the grub.cfg from its /, then boots Arch - is linuxefi correct here?

Just before GRUB2 drops me to the emergency shell I can type keys and they appear on the screen. When the shell opens the typed characters are still in the buffer and are entered in that shell.

Answer 1

Start Your system with the rd.debug (init-ramdisk debug) kernel flag added. This should shed some light on what is going wrong.
If You have it You kan bost the last one or two screens with xtrace output if You cannot the the problem Yourself then.

You can also direct the output to files in addition (i.e. add rd.log=all flag to kernel options) and should be able to get the log as copyable and better scrollable text after boot finished.

And, what I forgot, in Your above snippets You anonymized for whatever reasons the UUID and I don't know what is Your original config, but You have forgoten the dashes.
Example from my system (the blkid is the command that is run to find the physical device from Your cryptdevice=UUID=21685fd6-f2e3-4037-8645-3957cff3568c:cryptolvm kernel option, the part between cryptodevice= up to the first colon will be searched with the below command):

[root@ArchTestVM ~]# blkid -lt "UUID=21685fd6-f2e3-4037-8645-3957cff3568c" -o device
/dev/nvme0n1p2
[root@ArchTestVM ~]# blkid -lt "UUID=21685fd6f2e3403786453957cff3568c" -o device
[root@ArchTestVM ~]# 

Answer 2

You say "GRUB2 drops me to a prompt because it cannot mount /dev/mapper/cryptolvm-arch to /" but that does not look like a GRUB prompt. The commands you're entering are not GRUB commands, but (Arch) Linux commands. You're already past EFI, and past GRUB, and apparently within Arch initramfs.

The prompt looks like an initramfs-based emergency shell prompt. GRUB's job is to load two things: a kernel and an initramfs file. Once GRUB has done that and turned over control to the kernel, GRUB's job is done. The cryptomount command tells GRUB to unlock an encrypted disk for that purpose, but GRUB has no way to pass this unlocked state, or even an encryption passphrase, to the kernel.

Once the kernel starts up, the initramfs must have all the necessary tools to unlock the disk encryption again, either by prompting for the passphrase again, or by having the passphrase stored within the initramfs file; since the initramfs file is stored on an encrypted disk in your case, this might be acceptable.

Also, note that you said "...because it cannot mount /dev/mapper/cryptolvm-arch to /". If the error message says it exactly like that, the problem might be that the initramfs is trying to mount directly to / for some reason. You cannot do that. Instead, the new root filesystem must be first mounted somewhere else (like /new_root) and then a special command within the initramfs will switch the new root filesystem into place.

Instead of looking at EFI and GRUB, you should perhaps focus on the contents of Arch initramfs file. It should be running precisely those two commands you're entering manually (perhaps replacing /dev/nvme0n1p4 with an UUID= syntax), but for some reason that is not happening. Find out why, and you'll find the cause of your problems.