2

How might I be repeatedly corrupting my superblocks on RAID1

I have had two occasions in the past six months of RAID1 root arrays becoming unusable because of problems with the superblock. Unfortunately the first time I did not take good notes, but there were problems with suprblocks and bad magic. Just like the most recent occurence, I was powering down some KVM VMs when the host froze, forcing a hard restart after which I booted to an intramfs prompt with the superblock magic errors.

I'm about ready to give up on trying to recover my installation and go back to what worked for me (Arch on mirrored LVM), but I would like to know why this is happening. Am I configuring my partitions in a bad way? Should I assume that without hardware RAID, any crash or forced restart will result in corrupted superblocks? Is this a result of a bug and if so is it likely to be in mdadm, the distro, KVM, or is there no way to guess without exhaustive testing?

in the current crash, the relevant errors seem to be:

md1: invalid bitmap file for superblock: bad magic
md1: failed to create bitmap (-22)

This is what I boot to

When I boot using a Gparted rescue disk, there is no md1, but there is an md125 (the smallest ID number of the three). It is not visible to the gparted app, but I can see it under /dev/md125 and if I run sudo mdadm --detail /dev/md125 I get output indicating that it is active, failed, and not started. Also, "superblock is consistent"

I also tried:

e2fsck -b 8193 /dev/sda2
e2fsck -b 8193 /dev/sdb2
e2fsck -b 32768 /dev/sda2
e2fsck -b 32768 /dev/sdb2

and each time I got output about "bad magic number in super-block"

Does this indicate that the corruption was replicated across the arrays? If so, can I protect against this?

details:

  1. This happened with two different pairs of drives, each pair purchased separately.
  2. The drives behaved fine under LVM for several months after the first corruption incident. When I did tear down that system (Arch), it was not because of any problem, I just wanted to try Ubuntu Server and RAID1 again.
  3. The computer is attached to a UPS to protect against power fluctuations, which I do get. Unfortunately the electricity is not grounded, but my understanding is that this matters for surges, which are not a big worry where I am.
  4. The problems happened with Ubuntu Server 18.04.2, in the interim I used Arch with LVM on the same drives and had no problems.
  5. I have run memtest for a few passes and got no errors
  6. I have ECC RAM (unbuffered)
  7. System is a Ryzen Threadripper 1950x on an x399 Taichi motherboard.

My whole point was to maximize uptime by providing resilience against drive failures, but instead I seem to have decreased it because somehow the RAID1 superblocks get corrupted although the drives are probably physically OK. Is this an inherent risk in booting from RAID1?

Edit: also if I run mdadm --detail /dev/mdX,I get the following: mdadm from initramfs shell

Edit2: also, in initramfs, /etc/mdadm/mdadm.conf looks like this: mdadm.conf in initramfs

Edit3: The problem seems to be intermittent. I booted into Gparted and was able to see the problem md device once, and it came up clean, but after reboot was degraded again.

Edit4: it seems I can make the device come back and appear healthy by stopping the array in gparted and then rebooting into gparted (but still broken if I try to boot my installation)

Edit5: The above was a false alarm. It was just that md125 was now one of my healthy partitions, while the problem ones had become md126 and are still degraded.

Edit6: If I stop and assemble the array I get mdadm: failed to RUN_ARRAY /dev/md126: Invalid argument failed to RUN_ARRAY

  • 3
    you're not allowed to run fsck on raid member drives ... – frostschutz Jun 14 at 23:10
  • By "not allowed" you mean that it will harm something or just not work? – Stonecraft Jun 14 at 23:12
  • 2
    it might corrupt the raid superblock (which is different from filesystem superblock). even if it does work, it puts the raid out of sync while the raid layer is unaware, which can cause further corruption down the road. – frostschutz Jun 14 at 23:16
  • 2
    do you have mdadm --detail / --examine / /proc/mdstat when the raid is running? – frostschutz Jun 14 at 23:16
  • 1
    Should; sometimes doesn't. 40 yrs experience in this racket taught me. Fluorescent lights make 'white' light by the Hg vapor emitting UV, which hits a fluorescing coating inside the tube, that glows 'white' edisontechcenter.org/Fluorescent.html. When power cuts out, the coating glows for a fraction of a second after, so hoomans to sense short outages. Plus, our optic nerves clock at about 1/13 second at best newscientist.com/article/dn9633-calculating-the-speed-of-sight so you won'd see an outage of < 75ms. Combine the two, and yeah, you won't notice. – K7AAY Jun 17 at 16:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.