systemd-boot cannot find my root

Question

I'm running Debian Testing (Buster) and I'm swapping from Grub2 to systemd as I couldn't get Grub2 to work and someone suggested I try systemd-boot instead.

The boot/root drive is on an eMMC drive on the motherboard, whilst the data drive is on an mSATA SSD.

I have systemd-boot half working and it crashes on boot with this message

Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
done.
Gave up waiting for root file system device. Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT! PARTUUID=2251a5a4-6c18-425c-9264-df971d297b09 does not exist. Dropping to shell!

I've rebooted and managed to login using SuperGrub USB disk and I can see that the UUID does match my root partition so i don't know why it cannot find it.

/proc/cmdline output

BOOT_IMAGE=/boot/vmlinuz-4.19.0-5-amd64 root=UUID=2251a5a4-6c18-425c-9264-df971d297b09 ro

/boot and /boot/efi listings

kodi@BB-8:~$ ls /boot
config-4.15.0-3-amd64    grub                         System.map-4.15.0-3-amd64    vmlinuz-4.19.0-5-amd64
config-4.19.0-5-amd64    initrd.img-4.15.0-3-amd64    System.map-4.19.0-5-amd64    vmlinuz-4.9.45-ubilinux+
config-4.9.45-ubilinux+  initrd.img-4.19.0-5-amd64    System.map-4.9.45-ubilinux+
efi                      initrd.img-4.9.45-ubilinux+  vmlinuz-4.15.0-3-amd64

kodi@BB-8:~$ sudo ls /boot/efi
debian  EFI  loader

kodi@BB-8:~$ sudo ls /boot/efi/debian
drwx------ 2 root root     4096 Jul 16 22:31 .
drwx------ 5 root root     4096 Jan  1  1970 ..
-rwx------ 1 root root 31595838 Jul 26 11:09 initrd.img-4.15.0-3-amd64
-rwx------ 1 root root 33228805 Jul 26 11:09 initrd.img-amd64
-rwx------ 1 root root  4933392 Jul 26 11:09 vmlinuz-4.15.0-3-amd64
-rwx------ 1 root root  5217520 Jul 26 11:09 vmlinuz-amd64

UUID of drives and a df command

kodi@BB-8:~$ ls -al /dev/disk/by-uuid/
total 0
drwxr-xr-x 2 root root 220 Jul 26 08:59 .
drwxr-xr-x 8 root root 160 Jul 26 08:58 ..
lrwxrwxrwx 1 root root  15 Jul 26 11:10 2251a5a4-6c18-425c-9264-df971d297b09 -> ../../mmcblk0p2
lrwxrwxrwx 1 root root  10 Jul 26 11:10 42a36b04-83f8-4105-aef4-7f24b9ffff66 -> ../../sdb3
lrwxrwxrwx 1 root root  10 Jul 26 11:10 8280cf20-b70e-44f4-b092-6d3f92d54eab -> ../../dm-0
lrwxrwxrwx 1 root root  10 Jul 26 11:10 8A84-E6C0 -> ../../sdb2
lrwxrwxrwx 1 root root  10 Jul 26 11:10 8B91-8099 -> ../../sdb4
lrwxrwxrwx 1 root root  15 Jul 26 11:10 A9CE-4035 -> ../../mmcblk0p1
lrwxrwxrwx 1 root root  10 Jul 26 11:10 b2a67d10-07da-4eb4-bc16-b768084db045 -> ../../sda1
lrwxrwxrwx 1 root root  10 Jul 26 11:10 b618f5b0-2b8a-4e33-b288-407fd4355f83 -> ../../sdb5
lrwxrwxrwx 1 root root  10 Jul 26 11:10 f9a00ae7-07d2-4726-947b-03a4074049dd -> ../../sda2

kodi@BB-8:~$ df -h
Filesystem              Size  Used Avail Use% Mounted on
udev                    3.9G     0  3.9G   0% /dev
tmpfs                   783M   78M  705M  10% /run
/dev/mmcblk0p2           57G   14G   41G  26% /
tmpfs                   3.9G   39M  3.8G   1% /dev/shm
tmpfs                   5.0M  4.0K  5.0M   1% /run/lock
tmpfs                   3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1               820G  555G  224G  72% /home
/dev/mmcblk0p1          511M   82M  429M  17% /boot/efi
/dev/sda2                96G   67G   24G  74% /home/hts
10.1.1.1:/media/backup  2.7T  2.3T  178G  93% /media/backup
tmpfs                   783M   44K  783M   1% /run/user/1001
/dev/dm-0               7.8G   36M  7.3G   1% /media/kodi/8280cf20-b70e-44f4-b092-6d3f92d54eab
/dev/sdb4               7.9G  4.8G  3.2G  61% /media/kodi/DATA
/dev/sdb3                14G   12G  1.7G  88% /media/kodi/boot

Update: add modules

Add to /etc/initramfs-tools/modules

mmc_core
mmc_block
sdhci
sdhci-pci

Then type for 4.19.0-5 kernel

sudo update-initramfs -u -k all

Rebooted and I get the same message.

UPDATE: my systemd loader.conf

Here is my /boot/efi/loader/entries/debian.conf

title   Debian
linux   /debian/vmlinuz-amd64
initrd  /debian/initrd.img-amd64
options root=PARTUUID=2251a5a4-6c18-425c-9264-df971d297b09 rw

This is the UUID for my /dev/mmcblk0p2 eMMC device which is mounted as root.

Answer 1

On giving this another look, it seems you have a mismatch between PARTUUID=... and UUID=... and that's what's causing this problem.

You mentioned bootloader is configured with:

options root=PARTUUID=2251a5a4-6c18-425c-9264-df971d297b09 rw

But when you manage to boot it, you actually find this UUID under /dev/disk/by-uuid:

kodi@BB-8:~$ ls -al /dev/disk/by-uuid/
lrwxrwxrwx 1 root root  15 Jul 26 11:10 2251a5a4-6c18-425c-9264-df971d297b09 -> ../../mmcblk0p2

Furthermore, that's even listed in /proc/cmdline of the successful boot (I'm assuming it's the one with SuperGrub USB):

/proc/cmdline output

BOOT_IMAGE=/boot/vmlinuz-4.19.0-5-amd64 root=UUID=2251a5a4-6c18-425c-9264-df971d297b09 ro

The two UUIDs are different. PARTUUID= is the UUID that will be found in the GPT partition table (that's why it's called "PART", because it's a property of the partition, recorded in the partition table), while UUID= is a UUID recorded in the filesystem (ext4, or xfs, or whichever filesystem you formatted the partition with) and Linux is able to read those while scanning the disks.

So, it looks like you need to fix your boot options to use UUID= instead of PARTUUID=, since the UUID you have is a filesystem UUID and not a partition UUID.

Edit file /boot/efi/loader/entries/debian.conf and replace the last line with:

options root=UUID=2251a5a4-6c18-425c-9264-df971d297b09 rw

That should fix your issue!

Make sure your /etc/fstab in the system also matches the correct tag.

You can also use the blkid command to inspect the UUIDs present in your partitions and filesystems. This might help you confirm that you have the correct kind of UUID.

For instance, using blkid -o export should display something like:

$ sudo blkid -o export
DEVNAME=/dev/mmcblk0p1
SEC_TYPE=msdos
LABEL=boot
UUID=9A8B-7C6D
TYPE=vfat
PARTUUID=abcd1234-01

DEVNAME=/dev/mmcblk0p2
UUID=2251a5a4-6c18-425c-9264-df971d297b09
TYPE=ext4
PARTUUID=abcd1234-01

...

That should help you see all UUIDs with a tag that is recognized by Linux.

Answer 2

The device where your root partition lives is /dev/mmcblk0, which is an SD card. Modules for these devices are typically not compiled into the kernel, but into external modules, so they need to be included into your initramfs (or initrd) so they can be loaded before your root filesystem can be mounted.

On Debian, you can rebuild your initrd.img file using the update-initramfs command.

But first, you need to update the configuration to make sure the modules you need are going to be included into the updated initramfs.

You can either configure that using the MODULES directive in /etc/initramfs-tools/initramfs.conf and you can also list modules to load in /etc/initramfs-tools/modules and in /usr/share/initramfs-tools/modules.d/*. See man initramfs.conf for more details.

The modules you need are mmc_core and mmc_block. It's possible that you'll also need sdhci and sdhci-pci, so consider listing those as well.

Once you get your config updated, regenerate the initramfs for the kernel version you're trying to boot, with:

$ sudo update-initramfs -u -k 4.19.0-5-amd64

Or, to regenerate the initramfs files for all kernels:

$ sudo update-initramfs -u -k all

Hopefully, this is all you need for the kernel to find the root partition while booting.

Answer 3

It seems there is some misunderstanding here so far. The missing modules that are mentioned in the output are not kernel modules, but boot loader modules. What I see is the boot loader having issues finding the kernel image itself, perhaps because it cannot cope with the device it is on. At this stage, kernel modules are irrelevant.

I am not familiar with systemd-boot at all, but would expect that any modules you have to add go somewhere in /boot/efi/ on the partition that the bootloader is installed on. This is typically a FAT32-partition on your boot device.