How might I be repeatedly corrupting my superblocks on RAID1
I have had two occasions in the past six months of RAID1 root arrays becoming unusable because of problems with the superblock. Unfortunately the first time I did not take good notes, but there were problems with suprblocks and bad magic. Just like the most recent occurence, I was powering down some KVM VMs when the host froze, forcing a hard restart after which I booted to an intramfs prompt with the superblock magic errors.
I'm about ready to give up on trying to recover my installation and go back to what worked for me (Arch on mirrored LVM), but I would like to know why this is happening. Am I configuring my partitions in a bad way? Should I assume that without hardware RAID, any crash or forced restart will result in corrupted superblocks? Is this a result of a bug and if so is it likely to be in mdadm, the distro, KVM, or is there no way to guess without exhaustive testing?
in the current crash, the relevant errors seem to be:
md1: invalid bitmap file for superblock: bad magic md1: failed to create bitmap (-22)
When I boot using a Gparted rescue disk, there is no md1, but there is an md125 (the smallest ID number of the three). It is not visible to the gparted app, but I can see it under /dev/md125 and if I run
sudo mdadm --detail /dev/md125 I get output indicating that it is active, failed, and not started. Also, "superblock is consistent"
I also tried:
e2fsck -b 8193 /dev/sda2 e2fsck -b 8193 /dev/sdb2 e2fsck -b 32768 /dev/sda2 e2fsck -b 32768 /dev/sdb2
and each time I got output about "bad magic number in super-block"
Does this indicate that the corruption was replicated across the arrays? If so, can I protect against this?
- This happened with two different pairs of drives, each pair purchased separately.
- The drives behaved fine under LVM for several months after the first corruption incident. When I did tear down that system (Arch), it was not because of any problem, I just wanted to try Ubuntu Server and RAID1 again.
- The computer is attached to a UPS to protect against power fluctuations, which I do get. Unfortunately the electricity is not grounded, but my understanding is that this matters for surges, which are not a big worry where I am.
- The problems happened with Ubuntu Server 18.04.2, in the interim I used Arch with LVM on the same drives and had no problems.
- I have run memtest for a few passes and got no errors
- I have ECC RAM (unbuffered)
- System is a Ryzen Threadripper 1950x on an x399 Taichi motherboard.
My whole point was to maximize uptime by providing resilience against drive failures, but instead I seem to have decreased it because somehow the RAID1 superblocks get corrupted although the drives are probably physically OK. Is this an inherent risk in booting from RAID1?
Edit3: The problem seems to be intermittent. I booted into Gparted and was able to see the problem md device once, and it came up clean, but after reboot was degraded again.
Edit4: it seems I can make the device come back and appear healthy by stopping the array in gparted and then rebooting into gparted (but still broken if I try to boot my installation)
Edit5: The above was a false alarm. It was just that md125 was now one of my healthy partitions, while the problem ones had become md126 and are still degraded.