2

I have a VM which has an attached raided device with fstab entry:

/dev/md127 /mnt/blah ext4 nofail 0 2

The raided disks are corrupted and during startup the unit entered emergency/maintence mode, which means only the local host user could exit this mode and start it up normally. During normal startup the following occurred in syslog:

systemd-fsck[1272]: /dev/md127 contains a file system with errors, check forced.
systemd-fsck[1272]: /dev/md127: Inodes that were part of a corrupted orphan linked list found.
systemd-fsck[1272]: /dev/md127: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
systemd-fsck[1272]: #011(i.e., without -a or -p options)
systemd-fsck[1272]: fsck failed with exit status 4.
systemd-fsck[1272]: Running request emergency.target/start/replace
systemd[1]: systemd-fsck@dev-md127.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: systemd-fsck@dev-md127.service: Failed with result 'exit-code'.
systemd[1]: Failed to start File System Check on /dev/md127.
systemd[1]: Dependency failed for /mnt/blah.
systemd[1]: Dependency failed for Provisioner client daemon.

My guess is that the OS goes to emergency/maintenance mode because of the corrupt raided disks:

systemctl --state=failed
  UNIT                           LOAD   ACTIVE SUB    DESCRIPTION                                            
● systemd-fsck@dev-md127.service loaded failed failed File System Check on /dev/md127

What i want is for the VM to startup regardless of whether the raided drives are corrupt/unmountable, so it shouldn't go to emergency/maintenance mode. I followed these posts to attempt at disabling emergency/maintenance mode:

I had to first create the directory local-fs.target.d in /etc/systemd/system/, which felt wrong. I then created a nofail.conf in /etc/systemd/system/local-fs.target.d/nofail.conf containing:

[Unit]
OnFailure=

After loading that drop file, I was able to confirm that the drop file was found by local-fs.target:

sudo systemctl status local-fs.target 
● local-fs.target - Local File Systems
   Loaded: loaded (/lib/systemd/system/local-fs.target; static; vendor preset: enabled)
  Drop-In: /etc/systemd/system/local-fs.target.d
           └─nofail.conf
   Active: active since Tue 2019-01-08 12:36:41 UTC; 3h 55min ago
     Docs: man:systemd.special(7)

BUT, after rebooting, the VM still ended up in emergency/maintenance mode. Have i missed something? Does the nofail.conf solution not work with raided disks?


EDIT: I was able to get a print out of the logs when the system booted to emergency mode (sorry it's a screenshot since i don't have access to the host and had to ask the owner for it):

enter image description here

Here's the output from systemctl for systemd-fsck@dev-md127:

 sudo systemctl status --no-pager --full systemd-fsck@dev-md127
 ● systemd-fsck@dev-md127.service - File System Check on /dev/md127
    Loaded: loaded (/lib/systemd/system/systemd-fsck@.service; static; vendor preset: enabled)
    Active: failed (Result: exit-code) since Thu 2019-01-10 12:05:44 UTC; 2h 57min ago
      Docs: man:systemd-fsck@.service(8)
   Process: 1025 ExecStart=/lib/systemd/systemd-fsck /dev/md127 (code=exited, status=1/FAILURE)
  Main PID: 1025 (code=exited, status=1/FAILURE)

 systemd[1]: Starting File System Check on /dev/md127...
 systemd-fsck[1025]: /dev/md127 contains a file system with errors, check forced.
 systemd-fsck[1025]: /dev/md127: Inodes that were part of a corrupted orphan linked list found.
 systemd-fsck[1025]: /dev/md127: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
 systemd-fsck[1025]:         (i.e., without -a or -p options)
 systemd-fsck[1025]: fsck failed with exit status 4.
 systemd-fsck[1025]: Running request emergency.target/start/replace
 systemd[1]: systemd-fsck@dev-md127.service: Main process exited, code=exited, status=1/FAILURE
 systemd[1]: systemd-fsck@dev-md127.service: Failed with result 'exit-code'.
 systemd[1]: Failed to start File System Check on /dev/md127.

As i pointed out earlier, i have nofail set in /etc/fstab. Now the questions are:

  1. What is the dependency in the failed dependency in the screenshot?
  2. If fsck fails on /dev/md127, why does it enter emergency mode and how do i disable that?

EDIT 2:

A couple of other things i can add are:

  1. the vm is a kvm vm
  2. it's a software raid

Kind regards, Ankur

0

Your system tried to repair the corruption and boot automatically, but it could not. That is why it went to emergency mode.

This is not something that you can just override, because the filesystem cannot be mounted in this state, and you specified to mount the filesystem on boot.

You can try one of the following:

  • Don't corrupt the filesystem. Unmount it normally instead of hard powering off the VM while data are still being written to it.
  • Don't mount the filesystem automatically (set noauto in /etc/fstab). The VM will start, but you will still have to manually come in later and mount the filesystem.
  • Switch to a more resilient filesystem, such as XFS.
  • Hi Michael, thanks for the reply. This isn't a critical drive, and i've added nofail option in fstab, so it should boot normally. If it doesn't even boot so that ssh daemon starts, then it becomes extremely difficult to fix this problem in a timely manor. I believe that this option should work. I've tried the same setup with virtualbox and the VM boots up normally even if there are disk errors or the disk is not present, so now it seems that it could be a KVM issue. – Ankur22 Jan 17 at 17:14
  • @Ankur22 That is not what nofail does. It means only that the device will be ignored if it does not exist. If the device exists, this does nothing. – Michael Hampton Jan 17 at 17:16
  • That's interesting. The wording on the man page for systemd.mount suggest something else nofail With nofail, this mount will be only wanted, not required, by local-fs.target or remote-fs.target. This means that the boot will continue even if this mount point is not mounted successfully. whereas fstab does indeed suggest it will only ignore incase of missing device nofail do not report errors for this device if it does not exist. I know that there used to be a nobootwait option in 12.04, is there no similar option to avoid emergency mode? – Ankur22 Jan 17 at 17:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.