For a while I've been setting all servers up with software RAID 1 root and boot partitions in order to minimize downtime in case of disk failure. On install I set up
mdadm --create /dev/md0 --metadata 1.0 --raid-devices=2 --level=1 /dev/sd[ab]1
mdadm --create /dev/md1 --raid-devices=2 --level=1 /dev/sd[ab]2
mkfs.fat -F32 /dev/md0
mkfs.ext4 /dev/md1
and then set these up in fstab as /boot and /, respectively. Of course EFI can't read a software RAID 1 partition, so the EFI boot devices are /dev/sda1 and /dev/sdb1 individually. The initramfs is set up to know about mdadm, so / is immediately mounted as a /dev/md1 raid partition, and /dev/md0 is mirrored after boot, keeping /dev/sda1
and /dev/sdb1
synchronized. I've been setting systems up like this for years and have never had any problems.
Until recently, when I configured a backup server with 26 HBA disks. /dev/sdy
and /dev/sdz
are the SSD disks I used for /
and /boot
. Most of the time the system boots fine. Intermittently, however, on reboot the system crashes complaining that "the superblock on /dev/md1
is corrupted" and to "try running e2fsck -b 8198 etc.) to fix the problem. However, there is nothing wrong with the superblock on /dev/md1. I can boot from a USB stick and mount /dev/md1
with no problem, and e2fsck
returns no errors on the device. I'm at a loss for what could be causing this (other than intermittent hardware issues or perhaps flaky firmware). I briefly considered that it could be the large number of disks in the system combined with how mdadm
assembles raids by default (namely by scanning through partitions looking for RAID superblocks), but then realized this directive lives in /etc/mdadm.conf, which can't be read until after / is already mounted, which is the problem in the first place.
Has anyone else ever seen this problem or know of a solution?
mdadm.conf
is very likely copied to your initramfs. I'd suggest checkingcat /proc/mdstat
after the initramfs dies if possible (check if you can configure it to dump you to an emergency shell with busybox)—see if the array was actually assembled. Also look through the messages carefully, sometimes there are ten, fifteen lines after the actual failure hiding it.cat /proc/mdstat
when the system is in a failed to boot state, and both mdadm raids show up there. The boot partition is listed as/dev/md127
instead of/dev/md0
for some reason, but/dev/md1
is definitely there. I'm starting to think I have a firmware problem.