The goal is to have grub unlock /dev/nvme0n1p3 which contains a keyfile to unlock the 2 luks encrypted btrfs raid0 drives. If I can get it working, I'll create a tool that can accompany Linux installers to get it done easier.
I keep getting dropped into the grub rescue prompt with:
No such device: 2d6983f7-c10e-4b1a-b182-24d6f2b2a6c0
error: unknown filesystem.
So, it's not unlocking my luks. That's the UUID of /dev/mapper/cryptroot and /dev/mapper/cryptroot2 (They share it since it's raid0). Idk why it's showing up as the first thing grub tries to do though. The first thing I want grub to unlock is 0df41a34-e267-491a-ac02-25758c26ec65 aka /dev/nvme0n1p3 (cryptkeys) in order to unlock the raid0 drives. Here's what I did...
Setup
2 nvme drives.
- 2 NVMe drives.
- Garuda Linux (Arch-based).
- Grub 2.6 (Supports LUKS2).
blkidoutput:
/dev/loop1: TYPE="squashfs"
/dev/mapper/cryptroot2: UUID="2d6983f7-c10e-4b1a-b182-24d6f2b2a6c0" UUID_SUB="b2ee9dad-c9cb-4ec4-ae38-d28af19eb183" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/nvme0n1p3: UUID="0df41a34-e267-491a-ac02-25758c26ec65" TYPE="crypto_LUKS" PARTUUID="a49f7cdb-cbb6-44cd-b1e4-00b61dd1f00d"
/dev/nvme0n1p1: LABEL_FATBOOT="NO_LABEL" LABEL="NO_LABEL" UUID="A5AC-81DA" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="b0def085-1288-b746-9d7d-961354131dbc"
/dev/nvme0n1p2: UUID="802edb34-f481-4adf-9f98-3a80028d7cec" TYPE="crypto_LUKS" PARTLABEL="root" PARTUUID="9b945709-b51b-1c46-8ee3-6f3ba74c5a5b"
/dev/sdb2: SEC_TYPE="msdos" LABEL_FATBOOT="MISO_EFI" LABEL="MISO_EFI" UUID="EFD7-7387" BLOCK_SIZE="512" TYPE="vfat"
/dev/sdb1: BLOCK_SIZE="2048" UUID="2021-08-09-16-03-00-00" LABEL="GARUDA_GNOME_SOARING_" TYPE="iso9660"
/dev/loop2: TYPE="squashfs"
/dev/loop0: TYPE="squashfs"
/dev/mapper/cryptroot: UUID="2d6983f7-c10e-4b1a-b182-24d6f2b2a6c0" UUID_SUB="ef6be59d-a4be-4d00-93c2-0084530bf929" BLOCK_SIZE="4096" TYPE="btrfs"
/dev/nvme1n1: UUID="53517d3d-a638-48b9-af4f-125114e4f0c6" TYPE="crypto_LUKS"
/dev/zram0: LABEL="zram0" UUID="aa36a4d8-690e-4f2a-bfc9-e2fad1db8efb" TYPE="swap"
/dev/loop3: TYPE="squashfs"
Procedures
- Installed Garuda Linux to
/dev/nvme0n1which gave me the following partition layout on the first drive. I then created an ext4 partition (cryptkeys) in a luks container for storing keys and a luks container spanning the entire nvme1n1 for the btrfs raid:
NAME FSTYPE FLAGS
nvme0n1
├─nvme0n1p1 fat32 boot,esp
├─nvme0n1p2 crypto_LUKS
│ └─cryptroot btrfs
└─nvme0n1p3 crypto_LUKS
└─cryptkeys ext4
nvme1n1 crypto_LUKS
└─
└─cryptroot2 btrfs
Unlocked
nvme0n1p2andnvme1n1mounting to/mnt/cryptroot.To convert to raid0 spanning 2 drives, ran:
btrfs device add /dev/mapper/cryptroot2 /mnt/cryptroot
btrfs balance start -dconvert=raid0 -mconvert=raid1 /mnt/cryptroot
- Created a new keyfile for luks and added it to all luks containers except the one I named "cryptkeys" which is
/dev/nvme0n1p3. All luks containers can also be unlocked via the same password.nvme0n1p3was mounted to/mnt/cryptkeysand the keyfile copied to it:
dd bs=512 count=4 if=/dev/random of=/mnt/cryptroot/crypto_keyfile.bin
chmod 600 /mnt/cryptkeys/crypto_keyfile.bin
cryptsetup luksAddKey /dev/nvme0n1p2 cryptkeys/crypto_keyfile.bin
cryptsetup luksAddKey /dev/nvme1n1 cryptkeys/crypto_keyfile.bin
- With the btrfs raid0 now mounted, chrooted into the new Garuda install via:
mkdir /mnt/newroot
mount -o subvol=@,compress=zstd /dev/mapper/cryptroot newroot
for i in /dev /dev/pts /proc /sys /run; do sudo mount --bind $i /mnt/newroot$i; done
mount /dev/nvme0n1p1 newroot/boot/efi
mount --bind /sys/firmware/efi/efivars newroot/sys/firmware/efi/efivars
chroot /mnt/newroot
- Edited
/etc/default/grubto be:
# GRUB boot loader configuration
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Garuda"
GRUB_CMDLINE_LINUX_DEFAULT="quiet cryptdevice2=/dev/disk/by-uuid/0df41a34-e267-491a-ac02-25758c26ec65:cryptkeys:allow-discards cryptdevice3=/dev/disk/by-uuid/802edb34-f481-4adf-9f98-3a80028d7cec:cryptroot:allow-discards cryptdevice=/dev/disk/by-uuid/53517d3d-a638-48b9-af4f-125114e4f0c6:cryptroot2:allow-discards root=/dev/mapper/cryptroot splash rd.udev.log_priority=3 vt.global_cursor_default=0 systemd.unified_cgroup_hierarchy=1 loglevel=3"
GRUB_CMDLINE_LINUX=""
# Preload both GPT and MBR modules so that they are not missed
GRUB_PRELOAD_MODULES="part_gpt part_msdos"
# Uncomment to enable booting from LUKS encrypted devices
#GRUB_ENABLE_CRYPTODISK=y
# Set to 'countdown' or 'hidden' to change timeout behavior,
# press ESC key to display menu.
GRUB_TIMEOUT_STYLE=menu
# Uncomment to use basic console
GRUB_TERMINAL_INPUT=console
# Uncomment to disable graphical terminal
#GRUB_TERMINAL_OUTPUT=console
# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
GRUB_GFXMODE=auto
# Uncomment to allow the kernel use the same resolution used by grub
GRUB_GFXPAYLOAD_LINUX=keep
# Uncomment if you want GRUB to pass to the Linux kernel the old parameter
# format "root=/dev/xxx" instead of "root=/dev/disk/by-uuid/xxx"
#GRUB_DISABLE_LINUX_UUID=true
# Uncomment to disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY=true
# Uncomment and set to the desired menu colors. Used by normal and wallpaper
# modes only. Entries specified as foreground/background.
#GRUB_COLOR_NORMAL="light-blue/black"
#GRUB_COLOR_HIGHLIGHT="light-cyan/blue"
# Uncomment one of them for the gfx desired, a image background or a gfxtheme
#GRUB_BACKGROUND="/path/to/wallpaper"
GRUB_THEME="/usr/share/grub/themes/garuda/theme.txt"
# Uncomment to get a beep at GRUB start
#GRUB_INIT_TUNE="480 440 1"
# Uncomment to make GRUB remember the last selection. This requires
# setting 'GRUB_DEFAULT=saved' above.
#GRUB_SAVEDEFAULT=true
# Uncomment to disable submenus in boot menu
#GRUB_DISABLE_SUBMENU=y
GRUB_DISABLE_OS_PROBER=false
GRUB_DISABLE_OS_PROBER=false
GRUB_ENABLE_CRYPTODISK=y
- Copied hooks as:
# copy the original hook
cp /usr/lib/initcpio/install/encrypt /etc/initcpio/install/encrypt2
cp /usr/lib/initcpio/install/encrypt /etc/initcpio/install/encrypt3
cp /usr/lib/initcpio/hooks/encrypt /etc/initcpio/hooks/encrypt2
cp /usr/lib/initcpio/hooks/encrypt /etc/initcpio/hooks/encrypt3
# adapt the new hook to use different names and to NOT delete the keyfile
sed -i "s/cryptdevice/cryptdevice2/" /etc/initcpio/hooks/encrypt2
sed -i "s/cryptdevice/cryptdevice3/" /etc/initcpio/hooks/encrypt3
sed -i "s/cryptkey/cryptkey2/" /etc/initcpio/hooks/encrypt2
sed -i "s/cryptkey/cryptkey3/" /etc/initcpio/hooks/encrypt3
sed -i "s/rm -f \${ckeyfile}//" /etc/initcpio/hooks/encrypt2
sed -i "s/rm -f \${ckeyfile}//" /etc/initcpio/hooks/encrypt3
- Added
encrypt2andencrypt3to/etc/mkinitcpio.confbeforeencrypthook. Also specified keyfile.mkinitcpio.confis now:
# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run. Advanced users may wish to specify all system modules
# in this array. For instance:
# MODULES=(intel_agp i915 amdgpu radeon nouveau)
MODULES=(intel_agp i915 amdgpu radeon nouveau)
# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image. This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=()
# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way. This is useful for config files.
FILES="/crypto_keyfile.bin"
# HOOKS
# This is the most important setting in this file. The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added. Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
## This setup specifies all modules in the MODULES setting above.
## No raid, lvm2, or encrypted root is needed.
# HOOKS=(base)
#
## This setup will autodetect all modules for your system and should
## work as a sane default
# HOOKS=(base udev autodetect block filesystems)
#
## This setup will generate a 'full' image which supports most systems.
## No autodetection is done.
# HOOKS=(base udev block filesystems)
#
## This setup assembles a pata mdadm array with an encrypted root FS.
## Note: See 'mkinitcpio -H mdadm' for more information on raid devices.
# HOOKS=(base udev block mdadm encrypt filesystems)
#
## This setup loads an lvm2 volume group on a usb device.
# HOOKS=(base udev block lvm2 filesystems)
#
## NOTE: If you have /usr on a separate partition, you MUST include the
# usr, fsck and shutdown hooks.
HOOKS="base udev encrypt autodetect modconf block keyboard keymap consolefont plymouth encrypt2 encrypt3 encrypt filesystems"
# COMPRESSION
# Use this to compress the initramfs image. By default, zstd compression
# is used. Use 'cat' to create an uncompressed image.
#COMPRESSION="zstd"
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"
# COMPRESSION_OPTIONS
# Additional options for the compressor
#COMPRESSION_OPTIONS=()
- Ran:
mkinitcpio -p linux-zen
# initramfs includes the key, so only root should be able to read it
chmod 600 /boot/initramfs-linux-fallback.img
chmod 600 /boot/initramfs-linux.img
- Changed
/etc/crypttabto:
# /etc/crypttab: mappings for encrypted partitions.
#
# Each mapped device will be created in /dev/mapper, so your /etc/fstab
# should use the /dev/mapper/<name> paths for encrypted devices.
#
# See crypttab(5) for the supported syntax.
#
# NOTE: Do not list your root (/) partition here, it must be set up
# beforehand by the initramfs (/etc/mkinitcpio.conf). The same applies
# to encrypted swap, which should be set up with mkinitcpio-openswap
# for resume support.
#
# <name> <device> <password> <options>
cryptkeys UUID=0df41a34-e267-491a-ac02-25758c26ec65 /crypto_keyfile.bin luks,discard,nofail
- Changed
/etc/fstabto:
# <file system> <mount point> <type> <options> <dump> <pass>
UUID=A5AC-81DA /boot/efi vfat umask=0077 0 2
/dev/mapper/cryptroot / btrfs subvol=/@,defaults,noatime,space_cache,autodefrag,compress=zstd 0 0
/dev/mapper/cryptroot /home btrfs subvol=/@home,defaults,noatime,space_cache,autodefrag,compress=zstd 0 0
/dev/mapper/cryptroot /root btrfs subvol=/@root,defaults,noatime,space_cache,autodefrag,compress=zstd 0 0
/dev/mapper/cryptroot /srv btrfs subvol=/@srv,defaults,noatime,space_cache,autodefrag,compress=zstd 0 0
/dev/mapper/cryptroot /var/cache btrfs subvol=/@cache,defaults,noatime,space_cache,autodefrag,compress=zstd 0 0
/dev/mapper/cryptroot /var/log btrfs subvol=/@log,defaults,noatime,space_cache,autodefrag,compress=zstd 0 0
/dev/mapper/cryptroot /var/tmp btrfs subvol=/@tmp,defaults,noatime,space_cache,autodefrag,compress=zstd 0 0
- Finally, ran:
grub-mkconfig -o /boot/grub/grub.cfg
grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=Garuda --recheck
exit
reboot
An aside: A few times that I ran grub-install, the value of --bootloader-id was arch-grub before I changed it to Garuda. I don't think it matters much except that now I have extra boot menu entries as idk how to get rid of them. Probably doesn't matter though. I get the error even when selecting the Garuda entry from the EFI boot menu.
Note: These procedures were adapted from this blog post. What's different is no luks encrypted boot partition and the addition of the cryptkeys partition instead.
