mdadm + lvm2: destroyed all superblocks accidentally

Question

I accidentally overwrote all my RAID1 superblocks with garbage. I think this happened because I ALT-CTRL-DEL booted when the Ubuntu had put me in some kind of hard disk recovery mode. My display wasn't working and I did this "blind".

Looking at my RAID partitions, I see this:

# mdadm --examine /dev/sdc2
/dev/sdc2:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 00000000:00000000:00000000:00000000
  Creation Time : Fri Nov  1 18:59:05 2013
     Raid Level : -unknown-
   Raid Devices : 0
  Total Devices : 1
Preferred Minor : 127

    Update Time : Fri Nov  1 18:59:05 2013
          State : active
 Active Devices : 0
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 1
       Checksum : 6b1f0d22 - correct
         Events : 1


      Number   Major   Minor   RaidDevice State
this     0       8       34        0      spare   /dev/sdc2

   0     0       8       34        0      spare   /dev/sdc2

It is apparent that the superblocks have been fully overwritten by garbage. Both disks (sdb2, sdc2) look the same, the information is garbage, uuid is all zeros, raid level: unknown, raid devices: 0, etc.

The best bet I have is this:

mdadm --create --assume-clean --metadata=0.90 /dev/md0 --level=1 --raid-devices=2 /dev/sdb2 /dev/sdc2

Can I re-create my RAID array using mdadm --create like this?

On the RAID stack, I have an LVM2 physical volume. Can I somehow access my LVM2 data from the individual disks or backup disk images?

GRUB is able to find my initrd and kernel image from the disks, /boot is on ext4 root partition filesystem on top of LVM2, it is not a separate partition. So I believe the data is mostly intact, and the superblocks are gone.

edit: add --assume-clean to mdadm command line

Answer 1

If it's 0.90 superblock format, you should be able to just use a member directly (in read only mode if you please). That gives you access to your data and then you can create a new fresh RAID-1 with just the other disk, copy your data over, then add the original disk to the RAID.

If it's something else though (like 1.2 metadata) you'd first have to find the offset of the actual partition before you are able to mount it. For example if it's LVM you get the offset like this:

# grep -a -b --only-matching LEVELONE /dev/disk
1049088:LABELONE

Substract 512 since the first sector on a PV is free; if it actually was 0.90 or 1.0 metadata the grep should show 512: instead which means an actual offset of 0; if you get something larger, your RAID was using some other format.

In this case you can create a loop device with the correct offset:

# losetup -f --show --read-only --offset $((1049088-512)) /dev/disk
/dev/loop0
# file -s /dev/loop0
LVM2 ...

And that should be your PV and vgscan / vgchange -a y should take care of the rest.

---

As for mdadm --create, it can be done if you know what you're doing. You just have to be very, very careful with it. It's a last resort only.

If the offset is 0 make sure you use 0.90 metadata (if you're sure that's what you were using); if the offset is something else make sure the mdadm --create will use the same offset again. It usually doesn't as mdadm's defaults change over time. So specify it directly on the command line using --data-offset.

Also, create it with one disk only. Leave the other disk as missing. That's a better way to do it than --assume-clean as you can first check whether the create worked at all or not (is the data on the /dev/md? device usable?) and then have the other disk synced properly.