What are systemd *.device units actually waiting for?

Question

I've got a system that get's stuck when systemd tries to mount a filesystem. It fails on a dependency times out. The dependency is dev-disk-by\x2dlabel-cb\x2drootfs.device where the device being mounted is /dev/disk/by-label/cb-rootfs.

systemd[1]: dev-disk-by\x2dlabel-cb\x2drootfs.device: Job dev-disk-by\x2dlabel-cb\x2drootfs.device/start timed out.

Nothing stops me mounting the system with:

mount -av

So in this case it seems that systemd is waiting for the wrong thing, since the device is obviously there and ready to be mounted.

---

What might make this complex is that the root filesystem is an overlayfs involving the same device. In simple terms, initramfs is doing this:

mount /dev/disk/by-label/cb-rootfs /host
mount -o loop /host/rootfs/foo.squashfs /ro
mount -t overlay overlay lowerdir=/ro,upperdir=/host/writable,workdir=/host/workdir /root

/root is then the root file system ready for systemd to start mounting from /etc/fstab:

Then /etc/fstab tries to mount

/dev/disk/by-label/cb-rootfs /host ext4 defaults 0 0

---

As I say I can mount -av and Linux has absolutely no problem mounting `/dev/disk/by-label/cb-rootfs. But systemd is aparently waiting endlessly for it to become ready in some way.

What is systemd waiting for? how can I avoid it waiting like this?

Answer 1

In response to my question on systemd mailing lists I got this response:

Mantas Mikulėnas

.device units wait for udev to broadcast the uevent about that device being added, which happens after udev has 1. received the initial kernel uevent (either real or produced by systemd-udev-trigger.service) and 2. finished processing all its .rules for that device (which means everything that rules launched from RUN= must have exited, etc).

Only devices that udev rules have tagged with TAG+="systemd" will produce .device units; generally 99-systemd.rules will add that to disk devices.

If any of the rules have marked the device with ENV{SYSTEMD_READY}="0", the .device unit will keep waiting until another event removes that.

In my case it turned out udev was not installed properly or corrupted so despite the device being available in /dev systemd was still waiting on a signal from udev.

Simply reinstalling udev fixed the problem for me.