For a while I've been setting all servers up with software RAID 1 root and boot partitions in order to minimize downtime in case of disk failure. On install I set up
mdadm --create /dev/md0 --metadata 1.0 --raid-devices=2 --level=1 /dev/sd[ab]1 mdadm --create /dev/md1 --raid-devices=2 --level=1 /dev/sd[ab]2 mkfs.fat -F32 /dev/md0 mkfs.ext4 /dev/md1
and then set these up in fstab as /boot and /, respectively. Of course EFI can't read a software RAID 1 partition, so the EFI boot devices are /dev/sda1 and /dev/sdb1 individually. The initramfs is set up to know about mdadm, so / is immediately mounted as a /dev/md1 raid partition, and /dev/md0 is mirrored after boot, keeping
/dev/sdb1 synchronized. I've been setting systems up like this for years and have never had any problems.
Until recently, when I configured a backup server with 26 HBA disks.
/dev/sdz are the SSD disks I used for
/boot. Most of the time the system boots fine. Intermittently, however, on reboot the system crashes complaining that "the superblock on
/dev/md1 is corrupted" and to "try running e2fsck -b 8198 etc.) to fix the problem. However, there is nothing wrong with the superblock on /dev/md1. I can boot from a USB stick and mount /
dev/md1 with no problem, and
e2fsck returns no errors on the device. I'm at a loss for what could be causing this (other than intermittent hardware issues or perhaps flaky firmware). I briefly considered that it could be the large number of disks in the system combined with how
mdadm assembles raids by default (namely by scanning through partitions looking for RAID superblocks), but then realized this directive lives in /etc/mdadm.conf, which can't be read until after / is already mounted, which is the problem in the first place.
Has anyone else ever seen this problem or know of a solution?