Grub unlock luks encrypted btrfs raid0

Question

The goal is to have grub unlock /dev/nvme0n1p3 which contains a keyfile to unlock the 2 luks encrypted btrfs raid0 drives. If I can get it working, I'll create a tool that can accompany Linux installers to get it done easier.

I keep getting dropped into the grub rescue prompt with:

No such device: 2d6983f7-c10e-4b1a-b182-24d6f2b2a6c0
error: unknown filesystem.

So, it's not unlocking my luks. That's the UUID of /dev/mapper/cryptroot and /dev/mapper/cryptroot2 (They share it since it's raid0). Idk why it's showing up as the first thing grub tries to do though. The first thing I want grub to unlock is 0df41a34-e267-491a-ac02-25758c26ec65 aka /dev/nvme0n1p3 (cryptkeys) in order to unlock the raid0 drives. Here's what I did...

Setup

2 nvme drives.

• 2 NVMe drives.
• Garuda Linux (Arch-based).
• Grub 2.6 (Supports LUKS2).
• blkid output:

/dev/loop1: TYPE="squashfs"
/dev/mapper/cryptroot2: UUID="2d6983f7-c10e-4b1a-b182-24d6f2b2a6c0" UUID_SUB="b2ee9dad-c9cb-4ec4-ae38-d28af19eb183" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/nvme0n1p3: UUID="0df41a34-e267-491a-ac02-25758c26ec65" TYPE="crypto_LUKS" PARTUUID="a49f7cdb-cbb6-44cd-b1e4-00b61dd1f00d"
/dev/nvme0n1p1: LABEL_FATBOOT="NO_LABEL" LABEL="NO_LABEL" UUID="A5AC-81DA" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="b0def085-1288-b746-9d7d-961354131dbc"
/dev/nvme0n1p2: UUID="802edb34-f481-4adf-9f98-3a80028d7cec" TYPE="crypto_LUKS" PARTLABEL="root" PARTUUID="9b945709-b51b-1c46-8ee3-6f3ba74c5a5b"
/dev/sdb2: SEC_TYPE="msdos" LABEL_FATBOOT="MISO_EFI" LABEL="MISO_EFI" UUID="EFD7-7387" BLOCK_SIZE="512" TYPE="vfat"
/dev/sdb1: BLOCK_SIZE="2048" UUID="2021-08-09-16-03-00-00" LABEL="GARUDA_GNOME_SOARING_" TYPE="iso9660"
/dev/loop2: TYPE="squashfs"
/dev/loop0: TYPE="squashfs"
/dev/mapper/cryptroot: UUID="2d6983f7-c10e-4b1a-b182-24d6f2b2a6c0" UUID_SUB="ef6be59d-a4be-4d00-93c2-0084530bf929" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/nvme1n1: UUID="53517d3d-a638-48b9-af4f-125114e4f0c6" TYPE="crypto_LUKS"
/dev/zram0: LABEL="zram0" UUID="aa36a4d8-690e-4f2a-bfc9-e2fad1db8efb" TYPE="swap"
/dev/loop3: TYPE="squashfs"

Procedures

1. Installed Garuda Linux to /dev/nvme0n1 which gave me the following partition layout on the first drive. I then created an ext4 partition (cryptkeys) in a luks container for storing keys and a luks container spanning the entire nvme1n1 for the btrfs raid:

NAME               FSTYPE          FLAGS
nvme0n1
├─nvme0n1p1        fat32           boot,esp
├─nvme0n1p2        crypto_LUKS
│ └─cryptroot      btrfs
└─nvme0n1p3        crypto_LUKS
  └─cryptkeys      ext4
nvme1n1            crypto_LUKS
└─         
  └─cryptroot2     btrfs

2. Unlocked nvme0n1p2 and nvme1n1 mounting to /mnt/cryptroot.

3. To convert to raid0 spanning 2 drives, ran:

btrfs device add /dev/mapper/cryptroot2 /mnt/cryptroot
btrfs balance start -dconvert=raid0 -mconvert=raid1 /mnt/cryptroot

4. Created a new keyfile for luks and added it to all luks containers except the one I named "cryptkeys" which is /dev/nvme0n1p3. All luks containers can also be unlocked via the same password. nvme0n1p3 was mounted to /mnt/cryptkeys and the keyfile copied to it:

dd bs=512 count=4 if=/dev/random of=/mnt/cryptroot/crypto_keyfile.bin
chmod 600 /mnt/cryptkeys/crypto_keyfile.bin

cryptsetup luksAddKey /dev/nvme0n1p2 cryptkeys/crypto_keyfile.bin
cryptsetup luksAddKey /dev/nvme1n1 cryptkeys/crypto_keyfile.bin

5. With the btrfs raid0 now mounted, chrooted into the new Garuda install via:

mkdir /mnt/newroot
mount -o subvol=@,compress=zstd /dev/mapper/cryptroot newroot
for i in /dev /dev/pts /proc /sys /run; do sudo mount --bind $i /mnt/newroot$i; done
mount /dev/nvme0n1p1 newroot/boot/efi
mount --bind /sys/firmware/efi/efivars newroot/sys/firmware/efi/efivars 
chroot /mnt/newroot

6. Edited /etc/default/grub to be:

# GRUB boot loader configuration

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Garuda"
GRUB_CMDLINE_LINUX_DEFAULT="quiet cryptdevice2=/dev/disk/by-uuid/0df41a34-e267-491a-ac02-25758c26ec65:cryptkeys:allow-discards cryptdevice3=/dev/disk/by-uuid/802edb34-f481-4adf-9f98-3a80028d7cec:cryptroot:allow-discards cryptdevice=/dev/disk/by-uuid/53517d3d-a638-48b9-af4f-125114e4f0c6:cryptroot2:allow-discards root=/dev/mapper/cryptroot splash rd.udev.log_priority=3 vt.global_cursor_default=0 systemd.unified_cgroup_hierarchy=1 loglevel=3"
GRUB_CMDLINE_LINUX=""

# Preload both GPT and MBR modules so that they are not missed
GRUB_PRELOAD_MODULES="part_gpt part_msdos"

# Uncomment to enable booting from LUKS encrypted devices
#GRUB_ENABLE_CRYPTODISK=y

# Set to 'countdown' or 'hidden' to change timeout behavior,
# press ESC key to display menu.
GRUB_TIMEOUT_STYLE=menu

# Uncomment to use basic console
GRUB_TERMINAL_INPUT=console

# Uncomment to disable graphical terminal
#GRUB_TERMINAL_OUTPUT=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
GRUB_GFXMODE=auto

# Uncomment to allow the kernel use the same resolution used by grub
GRUB_GFXPAYLOAD_LINUX=keep

# Uncomment if you want GRUB to pass to the Linux kernel the old parameter
# format "root=/dev/xxx" instead of "root=/dev/disk/by-uuid/xxx"
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY=true

# Uncomment and set to the desired menu colors.  Used by normal and wallpaper
# modes only.  Entries specified as foreground/background.
#GRUB_COLOR_NORMAL="light-blue/black"
#GRUB_COLOR_HIGHLIGHT="light-cyan/blue"

# Uncomment one of them for the gfx desired, a image background or a gfxtheme
#GRUB_BACKGROUND="/path/to/wallpaper"
GRUB_THEME="/usr/share/grub/themes/garuda/theme.txt"

# Uncomment to get a beep at GRUB start
#GRUB_INIT_TUNE="480 440 1"

# Uncomment to make GRUB remember the last selection. This requires
# setting 'GRUB_DEFAULT=saved' above.
#GRUB_SAVEDEFAULT=true

# Uncomment to disable submenus in boot menu
#GRUB_DISABLE_SUBMENU=y

GRUB_DISABLE_OS_PROBER=false
GRUB_DISABLE_OS_PROBER=false
GRUB_ENABLE_CRYPTODISK=y

7. Copied hooks as:

# copy the original hook
cp /usr/lib/initcpio/install/encrypt /etc/initcpio/install/encrypt2
cp /usr/lib/initcpio/install/encrypt /etc/initcpio/install/encrypt3
cp /usr/lib/initcpio/hooks/encrypt  /etc/initcpio/hooks/encrypt2
cp /usr/lib/initcpio/hooks/encrypt  /etc/initcpio/hooks/encrypt3
# adapt the new hook to use different names and to NOT delete the keyfile
sed -i "s/cryptdevice/cryptdevice2/" /etc/initcpio/hooks/encrypt2
sed -i "s/cryptdevice/cryptdevice3/" /etc/initcpio/hooks/encrypt3
sed -i "s/cryptkey/cryptkey2/" /etc/initcpio/hooks/encrypt2
sed -i "s/cryptkey/cryptkey3/" /etc/initcpio/hooks/encrypt3
sed -i "s/rm -f \${ckeyfile}//" /etc/initcpio/hooks/encrypt2
sed -i "s/rm -f \${ckeyfile}//" /etc/initcpio/hooks/encrypt3

8. Added encrypt2 and encrypt3 to /etc/mkinitcpio.conf before encrypt hook. Also specified keyfile. mkinitcpio.conf is now:

# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES=(intel_agp i915 amdgpu radeon nouveau)
MODULES=(intel_agp i915 amdgpu radeon nouveau)

# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image.  This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=()

# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way.  This is useful for config files.
FILES="/crypto_keyfile.bin"

# HOOKS
# This is the most important setting in this file.  The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added.  Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
##   This setup specifies all modules in the MODULES setting above.
##   No raid, lvm2, or encrypted root is needed.
#    HOOKS=(base)
#
##   This setup will autodetect all modules for your system and should
##   work as a sane default
#    HOOKS=(base udev autodetect block filesystems)
#
##   This setup will generate a 'full' image which supports most systems.
##   No autodetection is done.
#    HOOKS=(base udev block filesystems)
#
##   This setup assembles a pata mdadm array with an encrypted root FS.
##   Note: See 'mkinitcpio -H mdadm' for more information on raid devices.
#    HOOKS=(base udev block mdadm encrypt filesystems)
#
##   This setup loads an lvm2 volume group on a usb device.
#    HOOKS=(base udev block lvm2 filesystems)
#
##   NOTE: If you have /usr on a separate partition, you MUST include the
#    usr, fsck and shutdown hooks.
HOOKS="base udev encrypt autodetect modconf block keyboard keymap consolefont plymouth encrypt2 encrypt3 encrypt filesystems"

# COMPRESSION
# Use this to compress the initramfs image. By default, zstd compression
# is used. Use 'cat' to create an uncompressed image.
#COMPRESSION="zstd"
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"

# COMPRESSION_OPTIONS
# Additional options for the compressor
#COMPRESSION_OPTIONS=()

9. Ran:

mkinitcpio -p linux-zen
# initramfs includes the key, so only root should be able to read it
chmod 600 /boot/initramfs-linux-fallback.img
chmod 600 /boot/initramfs-linux.img

10. Changed /etc/crypttab to:

# /etc/crypttab: mappings for encrypted partitions.
#
# Each mapped device will be created in /dev/mapper, so your /etc/fstab
# should use the /dev/mapper/<name> paths for encrypted devices.
#
# See crypttab(5) for the supported syntax.
#
# NOTE: Do not list your root (/) partition here, it must be set up
#       beforehand by the initramfs (/etc/mkinitcpio.conf). The same applies
#       to encrypted swap, which should be set up with mkinitcpio-openswap
#       for resume support.
#
# <name>               <device>                         <password> <options>
cryptkeys             UUID=0df41a34-e267-491a-ac02-25758c26ec65     /crypto_keyfile.bin luks,discard,nofail

11. Changed /etc/fstab to:

# <file system>             <mount point>  <type>  <options>  <dump>  <pass>
UUID=A5AC-81DA        /boot/efi      vfat    umask=0077 0 2
/dev/mapper/cryptroot /              btrfs   subvol=/@,defaults,noatime,space_cache,autodefrag,compress=zstd 0 0
/dev/mapper/cryptroot /home          btrfs   subvol=/@home,defaults,noatime,space_cache,autodefrag,compress=zstd 0 0
/dev/mapper/cryptroot /root          btrfs   subvol=/@root,defaults,noatime,space_cache,autodefrag,compress=zstd 0 0
/dev/mapper/cryptroot /srv           btrfs   subvol=/@srv,defaults,noatime,space_cache,autodefrag,compress=zstd 0 0
/dev/mapper/cryptroot /var/cache     btrfs   subvol=/@cache,defaults,noatime,space_cache,autodefrag,compress=zstd 0 0
/dev/mapper/cryptroot /var/log       btrfs   subvol=/@log,defaults,noatime,space_cache,autodefrag,compress=zstd 0 0
/dev/mapper/cryptroot /var/tmp       btrfs   subvol=/@tmp,defaults,noatime,space_cache,autodefrag,compress=zstd 0 0

12. Finally, ran:

grub-mkconfig -o /boot/grub/grub.cfg
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Garuda --recheck
exit
reboot

An aside: A few times that I ran grub-install, the value of --bootloader-id was arch-grub before I changed it to Garuda. I don't think it matters much except that now I have extra boot menu entries as idk how to get rid of them. Probably doesn't matter though. I get the error even when selecting the Garuda entry from the EFI boot menu.

Note: These procedures were adapted from this blog post. What's different is no luks encrypted boot partition and the addition of the cryptkeys partition instead.

Answer 1

Looks like you are not allowing GRUB to boot from LUKS encrypted devices:

# Uncomment to enable booting from LUKS encrypted devices

#GRUB_ENABLE_CRYPTODISK=y

Answer 2

I am planning to do a FDE installation with luks, lvm and btrfs. From what I could gather, GRUB is able to work with luks1 and not luks2. It may be a long shot, but you should check if Garuda installer created a luks2 partition, and if that is the case, downgrade it to luks1. Good luck and share your results!