0

For a while I've been setting all servers up with software RAID 1 root and boot partitions in order to minimize downtime in case of disk failure. On install I set up

mdadm --create /dev/md0 --metadata 1.0 --raid-devices=2 --level=1 /dev/sd[ab]1
mdadm --create /dev/md1 --raid-devices=2 --level=1 /dev/sd[ab]2
mkfs.fat -F32 /dev/md0
mkfs.ext4 /dev/md1

and then set these up in fstab as /boot and /, respectively. Of course EFI can't read a software RAID 1 partition, so the EFI boot devices are /dev/sda1 and /dev/sdb1 individually. The initramfs is set up to know about mdadm, so / is immediately mounted as a /dev/md1 raid partition, and /dev/md0 is mirrored after boot, keeping /dev/sda1 and /dev/sdb1 synchronized. I've been setting systems up like this for years and have never had any problems.

Until recently, when I configured a backup server with 26 HBA disks. /dev/sdy and /dev/sdz are the SSD disks I used for / and /boot. Most of the time the system boots fine. Intermittently, however, on reboot the system crashes complaining that "the superblock on /dev/md1 is corrupted" and to "try running e2fsck -b 8198 etc.) to fix the problem. However, there is nothing wrong with the superblock on /dev/md1. I can boot from a USB stick and mount /dev/md1 with no problem, and e2fsck returns no errors on the device. I'm at a loss for what could be causing this (other than intermittent hardware issues or perhaps flaky firmware). I briefly considered that it could be the large number of disks in the system combined with how mdadm assembles raids by default (namely by scanning through partitions looking for RAID superblocks), but then realized this directive lives in /etc/mdadm.conf, which can't be read until after / is already mounted, which is the problem in the first place.

Has anyone else ever seen this problem or know of a solution?

| improve this question | |
  • "but then realized this directive lives in /etc/mdadm.conf, which can't be read until after / is already mounted" mdadm.conf is very likely copied to your initramfs. I'd suggest checking cat /proc/mdstat after the initramfs dies if possible (check if you can configure it to dump you to an emergency shell with busybox)—see if the array was actually assembled. Also look through the messages carefully, sometimes there are ten, fifteen lines after the actual failure hiding it. – derobert Nov 22 '16 at 18:52
  • Thanks for the tips, especially the observation that mdadm.conf is being copied to initramfs (but of course -- why didn't I think of this?). After initramfs dies, if I try to <Ctrl>-d through to a prompt I get a kernel panic and the system is just frozen (so, in particular, I can't check /proc/mdstat). However, based on your input, let me modify /etc/mdadm.conf (I'm not sure how yet), and then rebuild the initramfs. I can usually get it to crash every 4-6 reboots, so it shouldn't be hard to test this hypothesis. Thanks again! – pgoetz Nov 22 '16 at 19:19
  • I'd not guess control-D to get a prompt—normally that'd give up on the prompt. What's it asking you that you're control-D'ing? At least the Debian one can be configured to eventually just dump you to a shell prompt. – derobert Nov 22 '16 at 19:26
  • I'll check, but I don't think I can do anything other than <Ctrl>-d from the "failed to mount / filesystem" prompt. <Ctrl>-d tells initramfs to continue the boot process anyway, which usually gets you to a functional prompt but in this case results in a kernel panic. – pgoetz Nov 22 '16 at 19:55
  • So, you were right. I can cat /proc/mdstat when the system is in a failed to boot state, and both mdadm raids show up there. The boot partition is listed as /dev/md127 instead of /dev/md0 for some reason, but /dev/md1 is definitely there. I'm starting to think I have a firmware problem. – pgoetz Nov 23 '16 at 20:00
1

I am using systemd-boot as my UEFI boot manager (https://wiki.archlinux.org/index.php/Systemd-boot), and systemd-boot requires that you provide loader configuration entries which provide information about which kernel/initramfs file to boot as well as options which are passed as kernel boot parameters. One of these options specifies the root device:

# cat /boot/loader/entries/arch.conf
title   Arch Linux
linux   /vmlinuz-linux
initrd  /initramfs-linux.img
options root=UUID=acf7e0be-5723-478e-aad6-80af6eb82a92 rw ipv6.disable=1

I had originally set the root device to be root=/dev/md1, as I normally do, but this system includes 26 HBA disks, and I think the system was getting confused about its disk device naming. In any case, changing the root to root=UUID=acf7e0be-5723-478e-aad6-80af6eb82a92 as illustrated above has resolved this issue, and the system now boots correctly every time.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.