1

Last weekend, I ran an apt-get dist-upgrade on a home Debian 8.3 system, and was surprised to see that it entered emergency mode on the next boot. The server in question runs LVM on top of a mdadm RAID 6 array. On closer inspection, it seemed to be struggling to mount the logical volumes:

Boot timeout

I've fumbled around for a week trying to fix the machine, but haven't really made any progress. I found this question which details an identical problem, but under different circumstances. I tried clean installing Debian, and found that if I don't let the installer download the latest versions of packages, it boots successfully the first time, then fails after a dist-upgrade. If I let it download them, the box restarts straight into emergency mode after the install. This is the entry in /var/log/apt/history.log for the former scenario; interestingly, it affects all of mdadm, udev and systemd:

Start-Date: 2016-01-30  19:48:28
Commandline: apt-get dist-upgrade
Upgrade: libpam-runtime:amd64 (1.1.8-3.1, 1.1.8-3.1+deb8u1), apt:amd64 (1.0.9.8.1, 1.0.9.8.2), multiarch-support:amd64 (2.19-18+deb8u1, 2.19-18+deb8u2), perl-base:amd64 (5.20.2-3+deb8u2, 5.20.2-3+deb8u3), libpam0g:amd64 (1.1.8-3.1, 1.1.8-3.1+deb8u1), apt-utils:amd64 (1.0.9.8.1, 1.0.9.8.2), libc-bin:amd64 (2.19-18+deb8u1, 2.19-18+deb8u2), libc6:amd64 (2.19-18+deb8u1, 2.19-18+deb8u2), mdadm:amd64 (3.3.2-5, 3.3.2-5+deb8u1), libapt-inst1.5:amd64 (1.0.9.8.1, 1.0.9.8.2), udev:amd64 (215-17+deb8u2, 215-17+deb8u3), base-files:amd64 (8+deb8u2, 8+deb8u3), libpam-modules:amd64 (1.1.8-3.1, 1.1.8-3.1+deb8u1), libudev1:amd64 (215-17+deb8u2, 215-17+deb8u3), libapt-pkg4.12:amd64 (1.0.9.8.1, 1.0.9.8.2), systemd-sysv:amd64 (215-17+deb8u2, 215-17+deb8u3), systemd:amd64 (215-17+deb8u2, 215-17+deb8u3), passwd:amd64 (4.2-3, 4.2-3+deb8u1), libpam-modules-bin:amd64 (1.1.8-3.1, 1.1.8-3.1+deb8u1), login:amd64 (4.2-3, 4.2-3+deb8u1), libsystemd0:amd64 (215-17+deb8u2, 215-17+deb8u3), libpcre3:amd64 (8.35-3.3, 8.35-3.3+deb8u2), locales:amd64 (2.19-18+deb8u1, 2.19-18+deb8u2), rsyslog:amd64 (8.4.2-1+deb8u1, 8.4.2-1+deb8u2)
End-Date: 2016-01-30  19:48:43

I'm really at a loss with this one. Can anyone offer any advice? As it's a clean install, I'm happy to experiment.

Improve this question

1 Answer 1

Reset to default
2

I had the same issue. The debian maintainers added a patch to mdadm which causes the raid assembly to start before the devices are up. I have not yet completely found out why as it's supposed to fix broken RAIDs as root file system.

But you can fix it for now by downgrading the mdadm package. Get the older version from here: http://snapshot.debian.org/package/mdadm/3.3.2-5/

Probably mdadm_3.3.2-5_amd64.deb for you. Install it with dpkg -i mdadm_3.3.2-5_amd64.deband then set it on hold until the patch is fixed. If you are using apt-get/apt use sudo apt-mark hold mdadm for aptitude its aptitude hold mdadm.

If you generally don't want your box to go into emergency mode (especially for mounts) put a nofail as option into the fstab. Also a good option is x-systemd.device-timeout so it doesn't wait 1:30min for a local device. An example fstab entry for the raid could look like this: /dev/md0 /media/md0 ext4 defaults,nofail,x-systemd.device-timeout=20 0 2

The bug is being tracked in #813335 and it also affects 3.3.4-1.1 from testing/unstable

Improve this answer
11
  • Is there a bug report open about this?
    – Faheem Mitha
    Commented Feb 2, 2016 at 8:12
  • Thanks @lhw. I'll try your suggestions after work and report back. If one's open, a bug report would be good to have - both to add that I ran into it, and to determine when it's safe to upgrade.
    – George Brighton
    Commented Feb 2, 2016 at 9:10
  • @FaheemMitha I ran into the issue yesterday and did not report the bug yet. I'll do it after work so that I can include logs and systemd unit states etc.
    – lhw
    Commented Feb 2, 2016 at 9:24
  • @GeorgeBrighton Are you using a raid card? Like LSI or HighPoint?
    – lhw
    Commented Feb 2, 2016 at 9:25
  • @lhw Software RAID only, but I'm using a Supermicro X10SL7-F, which has a LSI 2308 behind the 8 SATA ports - I don't know whether that's relevant? Two follow-up questions: 1) do you think it's better to downgrade to mdadm 3.3.2-5 rather than upgrade to 3.3.4-1.1? and 2) /home is also affected by the issue - presumably if I used nofail, I'd have login problems because my home directory wouldn't be mounted?
    – George Brighton
    Commented Feb 2, 2016 at 9:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .