systemd fails to umount manually mounted NFS shares in initramfs

Question

My diskless laboratory computers do not shutdown properly because systemd probably do not handle properly manual NFS mounts in initramfs. I'll describe shortly what is happening:

1. I created initramfs in which I mount ro several NFS shares (whole /, profiles, etc.) and overlay them using overlayfs to create rw Debian diskless for each workstation.
2. Then I switch_root to this overlayed system and it boots perfectly fine and works without problems. Homes are mounted separately during the boot based on fstab.
3. In my system I disabled Network Manager completely to prevent users from fiddling with network. This would be very harmful to them :)
4. When I'm doing shutdown system hangs with the nfs: server not responding, still trying... (this happens also with NM enabled, but in this case sometimes the shutdown succedes, but it is indeterministic).

So, my manual mount are "hidden" somehow from the systemd (I suspect that). My fstab:

none           /       overlay  defaults,x-systemd.requires=network-online.target 0 0
X.X.X.X:/home  /home   nfs      rw,nodev,nosuid,hard,nolock,proto=tcp,nfsvers=3,sec=krb5,x-systemd.requires=network-online.target    0   0

I tried _netdev, x-systemd.requires... but nothing helps.

Is there a solution for this? How can I umount all NFS shares under the overlayed / or at least forcibly reboot the system after /home is umounted properly? (system is diskless and all changes are in overlayfs, so it is safe to just, for example echo b > /proc/sysrq-trigger) ;)

Best Regards
Kamil

Answer 1

I spent two days and found somewhat reliable solution. I read half of the Internet and chatted on #systemd IRC. Please read this docs, maybe this will help you; for me, none of this helped, I couldn't make it work.

https://www.freedesktop.org/wiki/Software/systemd/InitrdInterface/
https://www.freedesktop.org/wiki/Software/systemd/RootStorageDaemons/

I ended up overriding systemd services responsible for shutting down or rebooting a machine.

Create override directories:

• /etc/systemd/system/systemd-poweroff.service.d
• /etc/systemd/system/systemd-reboot.service.d
• /etc/systemd/system/systemd-halt.service.d

And put inside override.conf files.
For poweroff:

[Unit]
SuccessAction=poweroff-immediate

For reboot:

[Unit]
SuccessAction=reboot-immediate

For halt (it is called for example, when Power Off button in Gnome is used):

[Unit]
ExecStart=systemctl --force --force halt

Adding a second --force or immediate does what it says. It does not wait for mounts or other things and just reboot/shutdown a machine. For NFS read-only mounts this is not harmful and can be done safely. Also, all changes from the overlay in my RAM will also be gone, which is fine. The /home, the only I care about will be umounted properly by the home.mount unit, created automatically for all fstab entries.

I hope that this will save someone two days of frustration ;)

Best Regards
Kamil