1

I messed up a bit and I am trying to find the best way to recover.

A few days ago, 1 of the physical disk of my lvm setup started to show sign of failure (I/O errors) so I decided to move it to another disk with pvmove. That didn't work out. After 5 days, pvmove had done only 0.1% so I stopped it.

After a reboot, the dying disk wouldn't show up at all, it had died completely so I decided to remove it with:

vgreduce --removemissing --force VolGroup00

Problem is that it refused to do so because of the pvmove saying the LV was locked. I tried:

pvmove --abort

But it refused to do so because of the missing disk that died.

So I was stuck and did:

vgcfgbackup VolGroup00

Then I edited the file, removed the entry about pvmove, tried:

vgcfgbackup VolGroup00

Which it refused to restore because of the missing disk so I edited the file again, removed the missing disk from there and did the vgcfgrestore which succeeded.

Now the problem is that I can't mount my volume because it says:

wrong fs type, bad option, bad superblock

Which makes sense as the size of the partition is supposed to be 2.4Tb but now has only 2.2Tb. Now the question is how do I fix this? Should I use a tool like testdisk or should I be able to somehow create a new physical volume / volume group where I can add my logical volumes which consist of 2 physical disks and somehow get the file system right (file system is ext4)?

pvdisplay output:

--- Physical volume ---
PV Name               /dev/sda4
VG Name               VolGroup00
PV Size               417.15 GiB / not usable 4.49 MiB
Allocatable           yes (but full)
PE Size               4.00 MiB
Total PE              106789
Free PE               0
Allocated PE          106789
PV UUID               dRhDoK-p2Dl-ryCc-VLhC-RbUM-TDUG-2AXeWQ

--- Physical volume ---
PV Name               /dev/sdb1
VG Name               VolGroup00
PV Size               1.82 TiB / not usable 4.97 MiB
Allocatable           yes (but full)
PE Size               4.00 MiB
Total PE              476923
Free PE               0
Allocated PE          476923
PV UUID               MF46QJ-YNnm-yKVr-pa3W-WIk0-seSr-fofRav 

I still have the config from before it died. Below is how it was. The disk that died (and which I removed) is pv1 (/dev/sdc1) but it doesn't want to restore this config because it says the disk is missing.

VolGroup00 {
    id = "a0p2ke-sYDF-Sptd-CM2A-fsRQ-jxPI-6sMc9Y"
    seqno = 4
    format = "lvm2"                 # informational
    status = ["RESIZEABLE", "READ", "WRITE"]
    flags = []
    extent_size = 8192              # 4 Megabytes
    max_lv = 0
    max_pv = 0
    metadata_copies = 0

    physical_volumes {

            pv0 {
                    id = "dRhDoK-p2Dl-ryCc-VLhC-RbUM-TDUG-2AXeWQ"
                    device = "/dev/sda4"    # Hint only

                    status = ["ALLOCATABLE"]
                    flags = []
                    dev_size = 874824678    # 417.149 Gigabytes
                    pe_start = 2048
                    pe_count = 106789       # 417.145 Gigabytes
            }

            pv1 {
                    id = "NOskcl-8nOA-PpZg-DCtW-KQgG-doKw-n3J9xd"
                    device = "/dev/sdc1"    # Hint only

                    status = ["ALLOCATABLE"]
                    flags = []
                    dev_size = 625142385    # 298.091 Gigabytes
                    pe_start = 2048
                    pe_count = 76311        # 298.09 Gigabytes
            }

            pv2 {
                    id = "MF46QJ-YNnm-yKVr-pa3W-WIk0-seSr-fofRav"
                    device = "/dev/sdb1"    # Hint only

                    status = ["ALLOCATABLE"]
                    flags = []
                    dev_size = 3906963393   # 1.81932 Terabytes
                    pe_start = 2048
                    pe_count = 476923       # 1.81932 Terabytes
            }
    }

    logical_volumes {

            lvolmedia {
                    id = "aidfLk-hjlx-Znrp-I0Pb-JtfS-9Fcy-OqQ3EW"
                    status = ["READ", "WRITE", "VISIBLE"]
                    flags = []
                    creation_host = "archiso"
                    creation_time = 1402302740      # 2014-06-09 10:32:20 +0200
                    segment_count = 3

                    segment1 {
                            start_extent = 0
                            extent_count = 476923   # 1.81932 Terabytes

                            type = "striped"
                            stripe_count = 1        # linear

                            stripes = [
                                    "pv2", 0
                            ]
                    }
                    segment2 {
                            start_extent = 476923
                            extent_count = 106789   # 417.145 Gigabytes

                            type = "striped"
                            stripe_count = 1        # linear

                            stripes = [
                                    "pv0", 0
                            ]
                    }
                    segment3 {
                            start_extent = 583712
                            extent_count = 76311    # 298.09 Gigabytes

                            type = "striped"
                            stripe_count = 1        # linear

                            stripes = [
                                    "pv1", 0
                            ]
                    }
            }
    }

Thank you for your help.

  • data in dead disk are lost. you can try to repair the filesystem using fsck -t ext4 /dev/myfs (assuming you have ext4 type filesystem, and /dev/myfs is your FS). did this fsck succeed ? if no, can you update with the error ? – Archemar Nov 13 '14 at 10:23
  • I just want to get the volume back with the 2 other disks. If I try to do fsck then I am getting: The filesystem size (according to the superblock) is 675863552 blocks The physical size of the device is 597721088 blocks Either the superblock or the partition table is likely to be corrupt! Abort<y>? So I abort, I need to somehow fix the partition table beforehand first. – ttux Nov 13 '14 at 13:00
0

I got farther in resolving my problem.

I came across this: https://www.novell.com/coolsolutions/appnote/19386.html#DiskPermanentlyRemoved

And so I plugged a new disk, did:

pvcreate --uuid NOskcl-8nOA-PpZg-DCtW-KQgG-doKw-n3J9xd --restorefile VolGroup00_00001-16738001.vg /dev/sdc1

The VolGroup00_00001-16738001.vg being the lvm config before the disk died and I removed it.

Then I did:

vgcfgrestore VolGroup00
vgscan
vgchange -ay VolGroup00

The VolGroup00 for vgcfgrestore being the VolGroup00_00001-16738001.vg of earlier. All of this was successful like on the novell link above.

I am able to mount the volume and get to my data now but I have the issue with fsck:

The filesystem size (according to the  superblock) is 675863552 blocks
The physical size of the device is  597721088 blocks
Either the superblock or the partition table is likely  to be corrupt!
Abort<y>?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.