LVM metadata missing, trying to recreate raid 1 with LVM

Question

I had a power issue in the house recently, and had issue getting my file server disks to mount. Turns out that one of the devices had renamed itself from sdb to sdd, and all of the LVM metadata is now missing. Using pvscan, lvscan, vgscan, etc all only show my system partition. Another reboot and the devices seemed to go back to they were before: sdb and sdc. I've managed to reassemble the raid using mdadm, but was unable to use vgcfgrestore to recreate my lvm configuration because apparently the UUID of my raid device has changed. My original VG was named "vg0". Here's the result of vgcfgrestore:

  Couldn't find device with uuid 3fgedF-F7Dc-c300-svuP-b3Q3-qSnb-CukkLq.
  Cannot restore Volume Group vg0 with 1 PVs marked as missing.
  Restore failed.

My /etc/lvm/backup/vg0 file shows this:

vg0 {
    id = "3JWsYl-FmEP-gpsa-7grO-VlLU-x7uC-EevgFc"
    seqno = 3
    format = "lvm2"         # informational
    status = ["RESIZEABLE", "READ", "WRITE"]
    flags = []
    extent_size = 8192      # 4 Megabytes
    max_lv = 0
    max_pv = 0
    metadata_copies = 0

    physical_volumes {

        pv0 {
            id = "3fgedF-F7Dc-c300-svuP-b3Q3-qSnb-CukkLq"
            device = "/dev/md0" # Hint only

            status = ["ALLOCATABLE"]
            flags = []
            dev_size = 3907028992   # 1.81935 Terabytes
            pe_start = 384
            pe_count = 476932   # 1.81935 Terabytes
        }
    }

    logical_volumes {

        data {
            id = "Sqjebo-rnKh-mgQH-a90E-Q0n7-idp1-1xPP56"
            status = ["READ", "WRITE", "VISIBLE"]
            flags = []
            segment_count = 1

            segment1 {
                start_extent = 0
                extent_count = 476932   # 1.81935 Terabytes

                type = "striped"
                stripe_count = 1    # linear

                stripes = [
                    "pv0", 0
                ]
            }
        }
    }
}

So the issue it seems I'm having is that the pv UUID is no longer valid, and I'm not even sure now what to use. The raid I managed to reassemble with --scan auto-named to /dev/md1, but even changing that in the vg0 backup file had no effect. I'm still not sure what the new pv UUID is.

# cat /proc/mdstat
Personalities : [raid1] 
md1 : active raid1 sdc1[1] sdb1[0]
      1953383488 blocks super 1.2 [2/2] [UU]
      bitmap: 0/15 pages [0KB], 65536KB chunk

unused devices: <none>

Again, pvs, lvs, and vgs all show only my root/system volumes and vg's, nothing from vg0. Any suggestions on next steps? Both drives are full of data (most of which is backed up) but I'd like to take whatever steps I can to save the filesystems.

EDIT:

Displaying the head of both disks (/dev/md1 shows garbage). I notice that only one of them has a LABELONE label:

[root@host ~]# head /dev/sdb1
üN+©Ûüþy {Gyì˧Rjedi:1RUYܯÜ1á×iSû«nZsH$ÊWYuQÿÿÿÿÿÿÿÿ>4þÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿvg0 {
id = "IwXCM3-LnxU-Oguo-PXiN-nXwq-VFaU-ZmgySs"
seqno = 1
format = "lvm2"
status = ["RESIZEABLE", "READ", "WRITE"]
flags = []
extent_size = 8192
max_lv = 0
max_pv = 0
metadata_copies = 0
[root@host ~]# head /dev/sdc1
LABELONEp­u+ LVM2 0013fgedFF7Dcc300svuPb3Q3qSnbCukkLqÁÑðüN+©Ûüþy {Gyì˧Rjedi:1RUYܯÜÒÆûPFlO!H$ÊWYuQÿÿÿÿÿÿÿÿ
ª9Úþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿþÿvg0 {
id = "IwXCM3-LnxU-Oguo-PXiN-nXwq-VFaU-ZmgySs"
seqno = 1
format = "lvm2"
status = ["RESIZEABLE", "READ", "WRITE"]
flags = []
extent_size = 8192
max_lv = 0
max_pv = 0
metadata_copies = 0

So now the 50 cent question: how do I recover the LVM labels without damaging the underlying filesystem?

UPDATE:

So I was basically able to successfully execute vgcfgrestore to a valid copy of my lvm backup config using a new PV UUID, and assembled /dev/md0 with that one drive, but now I'm getting a message that my PV is smaller than the allocated space. Basically it's reporting that my physical extents dropped from 476932 to 476900. The size of the disk hasn't changed, and I verified that the PV actually does have the correct number of extents available: (see the last line)

[root@host /]# pvs -v --segments /dev/md0
    Using physical volume(s) on command line.
    Wiping cache of LVM-capable devices
    Wiping internal VG cache
  Device /dev/md0 has size of 3906766976 sectors which is smaller than corresponding PV size of 3907028992 sectors. Was device resized?
  One or more devices used as PVs in VG vg0 have changed sizes.
  PV         VG   Fmt  Attr PSize PFree Start SSize  LV   Start Type   PE Ranges
  /dev/md0   vg0  lvm2 a--u 1.82t    0      0 476932 data     0 linear /dev/md0:0-476931

The last line shows that it's reporting extents from 0-476931, which is the correct size. I thought perhaps that the LVM headers itself may consume some space, but this isn't a new volume, it's been used for years without any issue and has never been resized. The volume is showing as suspended:

  LV Status              suspended
  # open                 0

I attempted to extend my PV with a USB thumbdrive (didn't think it would work, and it didn't) thinking if I could even temporarily mount this filesystem I could copy off the data and then create the whole raid from scratch, but of course that was not effective. Any thoughts on possible next steps to save the data?

Answer 1

First: head is not the best tool to display binary data. Try od or hexdump (Something like hexdump -C -n 4096 /dev/XYZ)

Second: This has nothing to do with md's id - LVM is using its own ids written in Physical Volume (PV) headers.

Third: It would be beneficial to post a tarball produced by lvmdump -sm (which contains e.g. /var/log/messages - so you may want to review its output.)

Some ideas:

Are these the only two disks there?

My first though was this looks like md was reassembled incorrectly - e.g. using wrong device overwriting one of your devices:

You are trying to restore vg0 with "UUID" "3JWsYl-FmEP-gpsa-7grO-VlLU-x7uC-EevgFc":

vg0 {
    id = "3JWsYl-FmEP-gpsa-7grO-VlLU-x7uC-EevgFc"

But on the md device's legs there is vg0 with different "UUID"

vg0 {
    id = "IwXCM3-LnxU-Oguo-PXiN-nXwq-VFaU-ZmgySs"

But the PV seems to have correct id:

    pv0 {
        id = "3fgedF-F7Dc-c300-svuP-b3Q3-qSnb-CukkLq"

vs. 3fgedFF7Dcc300svuPb3Q3qSnbCukkLq on one of the legs.

So I assume there is something else later in metadata area. For example: Is this a cloned vg and you changed its id later?

On second look it looks like one of the legs is shifted few bytes (or a part of the device was overwritten by zeros? That's why od/hexdump should be used). So md can not see anything else but rubbish - as the data on both disks do differ.

Were you manipulating partitions somehow? Updated kernel? Are you looking at the disks on different machine? This could be an alignment issue.

One of the legs seems to have correct PV header. LVM does not see it, as it is looking at md which returns junk. And LVM does not look at md's legs.

Possible Solutions

One possible solution is to disassemble the md into separate legs (Just remember: Do NOT zero superblock!) and let LVM look at the legs: run pvscan on the partitions - if the leg is correct, one of them might be fine.

Your metadata show there is only one linear LV having only one segment spanning over whole disk - that could be useful. What filesystem was on the device? If you have /etc/lvm/backup, I guess you have /etc/fstab too. As another possible solution is to find a start of FS and use dmsetup directly to create a mapping: https://wiki.gentoo.org/wiki/Device-mapper#Linear.

And last not least: try to keep the original devices read only.

Answer 2

So I ended up figuring out the issue on my own. I read somewhere that really old versions of mdadm used less metadata, and newer versions used more. Since I was moving from an Ubuntu 10.10 system to a CentOS 6.9 (Even though it had been successfully mounted on CentOS 6.9 for a few weeks) I figured that would explain why the /dev/md0 device was smaller than the original PV. Once I booted back up the Ubuntu 10.10 system, assembled the raid, and ran vgcfgrestore on the original volume group, the raid them mounted fine and my data was again available.

So basically raid filesystems built on really old versions of mdadm should not be mounted directly on newer distributions of Linux.