17

By default systemd drops to an emergency shell at the slightest error. For example, if one of the mounts at fstab fails for some reason the system becomes unbootable immediately. I manage dozens of diverse production systems and I've found this behavior very damaging. (Actually I think it's a major design failure, but that's a personal opinion).

I'd like to increase the system boot resilience. Optimally the system should always boot up, missing drivers, mounts, etc. shouldn't drop emergency shell, (just show warning instead) unless the given error would render console login absolutely impossible. What can be run, that should be run.

I know systemd automatically generates *.mount files from /etc/fstab and I could use the nofail option with small x-systemd.device timeout (or define the relevant .mount files myself). However it wouldn't solve my problem, I want to make the system more resilient, "patching" fstab every time is not very convenient and I'm not sure how many other possible "problems" exist which would render my system unbootable just because some developer somewhere thought it's important enough.

In sort, I'd like to regain the control over my machine and not let systemd decide what problem is serious enough to crush the boot process. Is it possible?

Improve this question
3
  • what's the actual problem btw? i'm aware of two - not being able to login over ssh, and the sulogin prompt only allowing root, not sudo users, to gain access in emergency mode. do those cover the damages you've suffered?
    – sourcejedi
    Jan 12, 2018 at 19:33
  • Actually the system would be much more accessible if those two services would be started, yes. Optimally the system should start everything that can be started just like in the old SysV times (error loggin' instead of painful death by emergency shell), and start the shell only in case of fatal error .
    – goteguru
    Jan 14, 2018 at 21:50
  • The actual problem (for me): "Cannot open access to console, the root account is locked." and "Press enter to continue." which starts it from the top.
    – tkteun
    Nov 19, 2020 at 17:11

3 Answers 3

Reset to default
11

It is literally only mount failures, that's all you would need to change.

So the letter of your request would be trivial to answer. Create a drop-in file:

# /etc/systemd/system/local-fs.target.d/nofail.conf

# Clear OnFailure= (set it to nothing)
[Unit]
OnFailure=

I believe this will add no new problem, beyond those that linux sysvinit already suffered by allowing this partial failure scenario.

However you also pointed out the question of how long systemd should wait for the specified block devices to become available. I can see no way to configure this, without providing a replacement for the fstab generator as a whole. https://www.freedesktop.org/software/systemd/man/systemd.generator.html

If you dump a large amount of less widely-used code here, it seems unlikely to increase system resilience. I think the closest solution would be to patch the existing fstab generator. It's not massively complex, I suspect you could get away with it / keep up with any significant changes.

Technically, if your distribution had a self-contained mountall sysvinit script, you could try hooking that in. But that will significantly change the boot process - it's actually more of a fork. I would not recommend that approach.

https://unix.stackexchange.com/a/393711/29483

If you search through the unit files, there are only a very few ways for the boot to fall back to emergency.target. It's usually when a .mount unit for a local filesystem fails, causing local-fs.target to fail. Or when your initramfs fails to mount the root filesystem, if your initramfs uses systemd.

local-fs.target has OnFailure=emergency.target. And it gets failed because units for local filesystems are automatically added to the Requires list of local-fs.target (unless they have DefaultDependencies=no).

$ systemctl show --property Requires local-fs.target
Requires=-.mount home.mount boot.mount boot-efi.mount
Improve this answer
6
  • 2
    I suppose I should put [Unit]\nOnFailure= into my nofail.conf. It seems to be possible to configure wait time in /etc/systemd/system.conf (via the generic DefaultTimeoutStartSec option). My systems are usually fast enough, 90s seems to be an overkill anyway. This solution seems to be promising.
    – goteguru
    Jan 14, 2018 at 22:03
  • In my case I set OnFailure= in /lib/systemd/system/local-fs.target instead of /etc/systemd (Ubuntu 16.04 on AWS)
    – ThiagoAlves
    Jul 30, 2019 at 2:35
  • 1
    @ThiagoAlves you should not do that, it will be overwritten on system upgrades. Follow the instructions in the answer, or ask for clarification :-).
    – sourcejedi
    Jul 30, 2019 at 7:15
  • 1
    @ThiagoAlves Thanks for your feedback. I've made the answer less ambiguous, so we can be clearer about whether that was the problem or not. I.e., I wonder if you made sure to include [Unit] before OnFailure=.
    – sourcejedi
    Aug 15, 2019 at 16:23
  • 1
    I had a CentOS 7 server going to emergency shell during boot for no reason (at least no errors in the serial console or systemd logs). Clearing the OnFailure option for local-fs.target did not help. By trial-and-error (sequentially clearing the OnFailure options for all unit files found by grep -r "OnFailure=" /usr/lib/systemd/) I found that I needed to clear the OnFailure option for initrd-parse-etc.service.
    – Quinn Comendant
    Sep 11, 2020 at 23:45
0

Turn off the automatic mounting of any filesystem that is non-essential to the boot operation by adding a noauto mount option to its /etc/fstab entry:

/dev/sdxy /u01 nfs defaults 0 0

to:

/dev/sdyx /u01 nfs noauto 0 0

and then mount the filesystem after boot by using a line in /etc/rc.local:

mount /u01

This example uses NFS but it also applicable to LUNs imported from a file server.

Improve this answer
2
  • 2
    Yes, I know noauto, but if I would change the fstab every time, nofail would be much better choice. Thx anyway.
    – goteguru
    Jan 14, 2018 at 21:50
  • That's an even more precise solution, but keep in mind that will still block boot for some given amount of time here. That can be adjusted too then, but at the end of the day I guess the real dilemma is just on which moment and situations you actually wish whatever overhead that may or may not be associated with mounting to happen.
    – mirh
    Nov 4, 2020 at 3:17
0

Try this maybe?

systemctl mask emergency.service
systemctl mask emergency.target
Improve this answer
1
  • 5
    Have you tried this? What happens when systemd encounters an error during boot, with the emergency target masked?
    – Stephen Kitt
    Feb 12, 2020 at 10:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .