Recover LVM (on RAID1): No volume groups found. Partition table erased?

Question

I'm trying to recover a LVM (no encryption) on a RAID1 using Debian Live.

Apparently, the RAID1 can be assembled without issue, but the LVM is broken. You may skip to LVM section. The RAID part is left in case it would matter.

The RAID

# aptitude install mdadm
# mdadm --assemble --scan

dmesg:

[  617.036709] md: md0 stopped.
[  617.038099] md: bind<sdc1>
[  617.038302] md: bind<sda1>
[  617.214903] md: raid1 personality registered for level 1
[  617.215534] md/raid1:md0: active with 2 out of 2 mirrors
[  617.215694] created bitmap (8 pages) for device md0
[  617.215956] md0: bitmap initialized from disk: read 1 pages, set 0 of
14903 bits
[  617.682354] md0: detected capacity change from 0 to 1000068874240
[  617.693821]  md0:

Here's the RAID:

# ls -l /dev/md0
brw-rw---- 1 root disk 9, 0 Jan 21 19:34 /dev/md0

# mdadm --examine /dev/md0
/dev/md0:
   MBR Magic : aa55

# file -s /dev/{md0,sda,sdc}
/dev/md0: DOS/MBR boot sector
/dev/sda: DOS/MBR boot sector
/dev/sdc: DOS/MBR boot sector

I'm afraid this DOS/MBR boot sector is the issue. More on this later.

Additional information, just in case

This is probably not relevant.

# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Sun Jun 21 18:04:33 2015
     Raid Level : raid1
     Array Size : 976629760 (931.39 GiB 1000.07 GB)
  Used Dev Size : 976629760 (931.39 GiB 1000.07 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Wed Jan 20 22:28:23 2016
          State : clean 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : bouzin:0
           UUID : 102b07b8:703e4597:574b2ecf:880a1aee
         Events : 4349

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       33        1      active sync   /dev/sdc1


# fdisk -l /dev/md0

Disk /dev/md0: 931.4 GiB, 1000068874240 bytes, 1953259520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x9c0ff432

# sfdisk -l /dev/md0

Disk /dev/md0: 244157440 cylinders, 2 heads, 4 sectors/track
Units: cylinders of 4096 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/md0p1          0       -       0          0    0  Empty
/dev/md0p2          0       -       0          0    0  Empty
/dev/md0p3          0       -       0          0    0  Empty
/dev/md0p4          0       -       0          0    0  Empty

# sfdisk -l /dev/sda

Disk /dev/sda: 121601 cylinders, 255 heads, 63 sectors/track
Units: cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sda1          0+ 121601- 121602- 976760832   fd  Linux raid
autodetect
/dev/sda2          0       -       0          0    0  Empty
/dev/sda3          0       -       0          0    0  Empty
/dev/sda4          0       -       0          0    0  Empty

# sfdisk -l /dev/sdc

Disk /dev/sdc: 121601 cylinders, 255 heads, 63 sectors/track
Units: cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdc1          0+ 121601- 121602- 976760832   fd  Linux raid
autodetect
/dev/sdc2          0       -       0          0    0  Empty
/dev/sdc3          0       -       0          0    0  Empty
/dev/sdc4          0       -       0          0    0  Empty

# cat /proc/mdstat
Personalities : [raid1] 
md0 : active (auto-read-only) raid1 sda1[0] sdc1[1]
      976629760 blocks super 1.2 [2/2] [UU]
      bitmap: 0/8 pages [0KB], 65536KB chunk

Edit 8: the RAID is mounted auto-read-only. As opposed to what I wrote here at first, I won't have to set it read-write, it will go read-write automatically when needed as explained here.

The LVM

# aptitude install lvm2

# pvscan
  No matching physical volumes found

# lvscan
  No volume groups found

Recover the configuration file

I don't have any backup of the lvm config file (I didn't know I needed to).

Following Recover Data From RAID1 LVM Partitions With Knoppix Linux LiveCD.

The idea is to read the start of the LVM partition to find the LVM config file.

# dd if=/dev/md0 bs=512 count=4096 skip=1 of=/tmp/md0-raw-start

# vi /tmp/md0-raw-start

Find the configuration file in there. Get rid of the binary and the older config versions.

Here is what I get (vg, lv,... are indeed the name I used when configuring the LVM):

# Generated by LVM2 version 2.02.111(2) (2014-09-01): Sun Jun 21 18:12:39 2015

contents = "Text Format Volume Group"
version = 1

description = ""

creation_host = "bouzin"        # Linux bouzin 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1 (2015-05-24) x86_64
creation_time = 1434910359      # Sun Jun 21 18:12:39 2015

vg {
id = "Yxknle-OEes-hihh-tWCt-QBxC-JtP9-bl360E"
seqno = 8
format = "lvm2"
status = ["RESIZEABLE", "READ", "WRITE"]
flags = []
extent_size = 8192
max_lv = 0
max_pv = 0
metadata_copies = 0

physical_volumes {

pv0 {
id = "gUyTdb-rc7j-rJh0-B2EZ-ebb7-mf77-KBgNWm"
device = "/dev/md0"

status = ["ALLOCATABLE"]
flags = []
dev_size = 1953259520
pe_start = 2048
pe_count = 238434
}
}

logical_volumes {

lv0 {
id = "AwliYc-HczW-LZ1x-czpO-YZOJ-sr7k-T13HUf"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_host = "bouzin"
creation_time = 1434910352
segment_count = 1

segment1 {
start_extent = 0
extent_count = 953

type = "striped"
stripe_count = 1

stripes = [
"pv0", 0
]
}
}

lv1 {
id = "Ec1tN2-WKaf-v2if-lAu2-MfiI-1hkE-XyKFGI"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_host = "bouzin"
creation_time = 1434910359
segment_count = 1

segment1 {
start_extent = 0
extent_count = 7152

type = "striped"
stripe_count = 1

stripes = [
"pv0", 953
]
}
}

lv2 {
id = "gFWdEh-7HUJ-zwX1-nqEU-DomC-tdfW-ZGChNw"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_host = "bouzin"
creation_time = 1434910366
segment_count = 1

segment1 {
start_extent = 0
extent_count = 230329

type = "striped"
stripe_count = 1

stripes = [
"pv0", 8105
]
}
}
}
}

This confirms I setup the partitions in this order:

swap
/
/home

Edit 2: Important note

As opposed to what is shown in the page I link to, don't miss the few lines before vg {, especially the contents = ..., otherwise you'll get

`Can't process text format file - missing contents field.` 

error when using vgcfgrestore.

Use recovered configuration file

Install recovered config file in lvm config directory and start lvm.

# mkdir /etc/lvm/backup
# cp /tmp/md0-raw-start /etc/lvm/backup/vg

# systemctl start lvm2

# systemctl status lvm2
● lvm2-activation.service - Activation of LVM2 logical volumes
   Loaded: loaded (/lib/systemd/system/lvm2-activation.service; enabled)
   Active: inactive (dead) since Thu 2016-01-21 20:37:42 UTC; 4s ago
     Docs: man:lvm(8)
           man:vgchange(8)
  Process: 22212 ExecStart=/sbin/lvm vgchange -aay --sysinit
(code=exited, status=0/SUCCESS)
 Main PID: 22212 (code=exited, status=0/SUCCESS)

Jan 21 20:37:42 debian lvm[22212]: No volume groups found

And here's the problem. No volume groups found.

# vgscan
  Reading all physical volumes.  This may take a while...
  No volume groups found

Nope.

# vgcfgrestore vg
  Couldn't find device with uuid gUyTdb-rc7j-rJh0-B2EZ-ebb7-mf77-KBgNWm.
  Cannot restore Volume Group vg with 1 PVs marked as missing.
  Restore failed.

Since I identified and fixed missing lines in my recovered lvm config file (see Edit 2 above), this error message from vgcfgrestore is more explicit.

Still, where do I go from here?

Partition table erased?

Back to the description of the array:

# file -s /dev/{md0,sda,sdc}
/dev/md0: DOS/MBR boot sector
/dev/sda: DOS/MBR boot sector
/dev/sdc: DOS/MBR boot sector

From another post here, I'd expect something like this:

$ file -s /dev/{sde1,md2}
/dev/sde1: LVM2 (Linux Logical Volume Manager) , UUID:
ZK8IfBzUHPH5befvm5CZ81oIXHm11TG
/dev/md2:  LVM2 (Linux Logical Volume Manager) , UUID:
ZK8IfBzUHPH5befvm5CZ81oIXHm11TG

Last boot before this issue, I installed Linux Mint using a USB stick on another machine, using this machine to create the bootable drive. I copied the "hybrid" .iso on the stick using dd then had issues formating it back to FAT32 using GParted. I think I tried a few fdisk then ultimately gave up.

Thinking of it, it is likely I messed up with my system using fdisk on the wrong /dev. I can't remember for sure what I did, but it could be a clue. I can't think of anything else. The system is Debian Jessie with unattended-upgrades, but I don't think an automatic update did this.

Note that the partition starts with the swap, so erasing the beginning could be less critical than if it started with important files.

Can someone confirm that DOS/MBR boot sector here is the issue and could be due to a USB stick partitioning mistake?

And most importantly, any idea how to fix this?

(I have daily backup of most of the important files on the drive. I'd like to solve this for the sake of understanding, and because I'd like to check the files I might miss backups for.)

Instructions here may apply, but I'd appreciate a bit more details before proceeding as it is a bit unclear to me.

Edit 1: Testdisk option

Someone suggests giving up the partition table recovery but using Testdisk to recover the data. I might try to backup existing data this way before trying anything heroic with pvcreate.

FWIW, here's the output of Testdisk.

Analysis

Disk /dev/md0 - 1000 GB / 931 GiB - CHS 244157440 2 4
Current partition structure:
     Partition                  Start        End    Size in sectors

No partition is bootable

Quick search

Disk /dev/md0 - 1000 GB / 931 GiB - CHS 244157440 2 4
     Partition               Start        End    Size in sectors
 * Linux Swap             255   0  1   256   1  4         16
 P Linux                976128   0  1 8299775   1  4   58589184
 P Linux                8299776   0  1 244156671   1  4 1886855168

SWAP2 version 0, pagesize=8192, 8192 B
ext4 blocksize=4096 Large file Sparse superblock, 29 GB / 27 GiB
ext4 blocksize=4096 Large file Sparse superblock, 966 GB / 899 GiB

Edit 2: pvcreate option

Here's the error message again.

# vgcfgrestore vg
  Couldn't find device with uuid gUyTdb-rc7j-rJh0-B2EZ-ebb7-mf77-KBgNWm.

Now, following this suggestion, should I try this?

dd if=/dev/zero count=1 of=/dev/md0
pvcreate --uuid gUyTdb-rc7j-rJh0-B2EZ-ebb7-mf77-KBgNWm --norestorefile
vcfgrestore

I'm pretty sure this is it, but I'd appreciate a confirmation.

Edit 3: Symptoms

I forgot to mention the error messages I got at startup.

First reboot, I had this error:

error: disk `lvmid/Yxknle-OEes-...` not found. 
Entering rescue mode...
grub rescue> ls
(hd0) (hdO,msdos1), (hd1) (hd1,msdos1) (hd2) (hd2,msdos2) (md/0)

Then I tried removing one disk. No change. Then the other and the error changed and I now consistently have this new error:

error: file `/boot/grub/i386-pc/normal.mod` not found.

Edit 4: strace pvscan

# strace -e trace=open pvscan 2>&1 | grep /dev/md
open("/dev/md0", O_RDONLY|O_DIRECT|O_NOATIME) = 3
open("/dev/md0", O_RDONLY)              = 5
open("/dev/md0", O_RDONLY)              = 3
open("/dev/md0", O_RDONLY|O_DIRECT|O_NOATIME) = 3
open("/dev/md0", O_RDONLY|O_DIRECT|O_NOATIME) = 3

Edit 5: lvm backup file recovered

Using Testdisk, I managed to get hands on /etc/lvm/backup/vg.

# Generated by LVM2 version 2.02.111(2) (2014-09-01): Sun Jun 21 20:19:54 2015

contents = "Text Format Volume Group"
version = 1

description = "Created *after* executing 'vgcfgbackup'"

creation_host = "bouzin"        # Linux bouzin 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1 (2015-05-24) x86_64
creation_time = 1434910794      # Sun Jun 21 20:19:54 2015

vg {
        id = "Yxknle-OEes-hihh-tWCt-QBxC-JtP9-bl360E"
        seqno = 8
        format = "lvm2"                 # informational
        status = ["RESIZEABLE", "READ", "WRITE"]
        flags = []
        extent_size = 8192              # 4 Megabytes
        max_lv = 0
        max_pv = 0
        metadata_copies = 0

        physical_volumes {

                pv0 {
                        id = "gUyTdb-rc7j-rJh0-B2EZ-ebb7-mf77-KBgNWm"
                        device = "/dev/md0"     # Hint only

                        status = ["ALLOCATABLE"]
                        flags = []
                        dev_size = 1953259520   # 931,387 Gigabytes
                        pe_start = 2048
                        pe_count = 238434       # 931,383 Gigabytes
                }
        }

        logical_volumes {

                lv0 {
                        id = "AwliYc-HczW-LZ1x-czpO-YZOJ-sr7k-T13HUf"
                        status = ["READ", "WRITE", "VISIBLE"]
                        flags = []
                        creation_host = "bouzin"
                        creation_time = 1434910352      # 2015-06-21 20:12:32 +0200
                        segment_count = 1

                        segment1 {
                                start_extent = 0
                                extent_count = 953      # 3,72266 Gigabytes

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = [
                                        "pv0", 0
                                ]
                        }
                }

                lv1 {
                        id = "Ec1tN2-WKaf-v2if-lAu2-MfiI-1hkE-XyKFGI"
                        status = ["READ", "WRITE", "VISIBLE"]
                        flags = []
                        creation_host = "bouzin"
                        creation_time = 1434910359      # 2015-06-21 20:12:39 +0200
                        segment_count = 1

                        segment1 {
                                start_extent = 0
                                extent_count = 7152     # 27,9375 Gigabytes

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = [
                                        "pv0", 953
                                ]
                        }
                }

                lv2 {
                        id = "gFWdEh-7HUJ-zwX1-nqEU-DomC-tdfW-ZGChNw"
                        status = ["READ", "WRITE", "VISIBLE"]
                        flags = []
                        creation_host = "bouzin"
                        creation_time = 1434910366      # 2015-06-21 20:12:46 +0200
                        segment_count = 1

                        segment1 {
                                start_extent = 0
                                extent_count = 230329   # 899,723 Gigabytes

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = [
                                        "pv0", 8105
                                ]
                        }
                }
        }
}

It is identical to what I had recovered, except it has comments.

Edit 6: trying to create Physical Volume

From above, /dev/md0 size is 976629760 kB.

# dd if=/dev/md0 of=/media/user/bak/copy_lvm/start bs=1M count=1
# dd if=/dev/md0 of=/media/user/bak/copy_lvm/end bs=1M count=1 skip=953739

(Hoping I'm using dd correctly.)

Not sure how I should use pvcreate:

# pvcreate --uuid gUyTdb-rc7j-rJh0-B2EZ-ebb7-mf77-KBgNWm --norestorefile
  Can only set uuid on one volume at once
  Run `pvcreate --help' for more information.

# pvcreate --uuid gUyTdb-rc7j-rJh0-B2EZ-ebb7-mf77-KBgNWm --norestorefile /dev/md0
  Can't open /dev/md0 exclusively.  Mounted filesystem?

I tried to set the array read-write.

# mdadm --readwrite /dev/md0
mdadm: failed to set writable for /dev/md0: Device or resource busy

I don't know why it is busy. lsof yields nothing.

# lsof /dev/md0

Edit 7: Testdisk again

Thanks to Testdisk, I managed to backup the files I had no automatic backup for. I should be safe, now. It's only about saving the system to avoid reinstalling from scratch. (I even copied /etc.)

Testdisk also does partition detection and partition table repair. It detected my partitions (see above) but indicated the swap was the bootable one. I changed this to the one holding the system. Perhaps some Testdisk trickery also happened behind the scene. Anyway, I clicked "Write", then rebooted.

Still, same error at reboot:

error: file `/boot/grub/i386-pc/normal.mod` not found.

However, there's good news: booting on Debian Live, the array is automatically assembled and the LVM recognized. I can browse the partitions.

I can also check that /boot/grub/i386-pc/normal.mod is where it should be. (It is binary so I can't check what's in it.)

Oh, and I also checked root's bash history, and I didn't find a command that would have caused this mess. I used fdisk on /dev/sdh, but not on /dev/sda or /dev/sdc by mistake. Could be with GParted, though.

Edit 8: RAID and LVM status

Since things have evolved, I thought I'd try those commands again.

# mdadm --examine /dev/md0
/dev/md0:
   MBR Magic : aa55
Partition[0] :           16 sectors at         2040 (type 82)
Partition[1] :     58589184 sectors at      7809024 (type 83)
Partition[2] :   1886855168 sectors at     66398208 (type 83)

# file -s /dev/{md0,sda,sdc}
/dev/md0: DOS/MBR boot sector; partition 1 : ID=0x82, start-CHS (0xff,0,1), end-CHS (0x10,1,4), startsector 2040, 16 sectors; partition 2 : ID=0x83, active, start-CHS (0x3ff,1,4), end-CHS (0x3ff,1,4), startsector 7809024, 58589184 sectors; partition 3 : ID=0x83, start-CHS (0x3ff,1,4), end-CHS (0x3ff,1,4), startsector 66398208, 1886855168 sectors
/dev/sda: DOS/MBR boot sector
/dev/sdc: DOS/MBR boot sector

# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Sun Jun 21 18:04:33 2015
     Raid Level : raid1
     Array Size : 976629760 (931.39 GiB 1000.07 GB)
  Used Dev Size : 976629760 (931.39 GiB 1000.07 GB)
   Raid Devices : 2
  Total Devices : 2
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Sat Jan 23 21:43:23 2016
          State : clean 
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           Name : bouzin:0
           UUID : 102b07b8:703e4597:574b2ecf:880a1aee
         Events : 4355

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       33        1      active sync   /dev/sdc1

# fdisk -l /dev/md0

Disk /dev/md0: 931.4 GiB, 1000068874240 bytes, 1953259520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x9c0ff432

Device     Boot    Start        End    Sectors   Size Id Type
/dev/md0p1          2040       2055         16     8K 82 Linux swap / Solaris
/dev/md0p2 *     7809024   66398207   58589184    28G 83 Linux
/dev/md0p3      66398208 1953253375 1886855168 899.7G 83 Linux

# sfdisk -l /dev/md0

Disk /dev/md0: 244157440 cylinders, 2 heads, 4 sectors/track
Units: cylinders of 4096 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/md0p1        255     256       2          8   82  Linux swap / Solaris
/dev/md0p2   * 976128  8299775  7323648   29294592   83  Linux
/dev/md0p3     8299776  244156671  235856896  943427584   83  Linux
/dev/md0p4          0       -       0          0    0  Empty

# cat /proc/mdstat
Personalities : [raid1] 
md0 : active raid1 sda1[0] sdc1[1]
      976629760 blocks super 1.2 [2/2] [UU]
      bitmap: 0/8 pages [0KB], 65536KB chunk

unused devices: <none>

# vgscan
  Reading all physical volumes.  This may take a while...
  No volume groups found

I think I'm beginning to understand your doubts. It all looks like the RAID (/dev/md0) is partitioned without LVM. This would be surprising, though, as I remember creating the LVM and the configuration file I found confirms it.

Could it be Testdisk ignored the LVM, then wrote the partition table straight to /dev/md0 sort of shunting the LVM (if that makes any sense)?

Edit 9: Where is my LVM

FWIW, I rebooted, still on Debian Live and after installing mdadm, the raid is automatically assembled even before lvm2 is installed (only liblvm2app2.2 is). Does this mean the LVM has "vanished"?

# dmsetup ls
No devices found

# pvscan
  No matching physical volumes found

# vgscan 
  Reading all physical volumes.  This may take a while...
  No volume groups found

Edit 10: Grub repair

Let's assume the filesystem/LVM work correctly and focus on the Grub error.

Following advices on the Internet, I tried this grub-install

# mount /dev/md0p2 /mnt
# grub-install --root-directory=/mnt /dev/md0
The file /mnt/boot/grub/stage1 not read correctly.

The partition is recognized as Linux:

# fdisk -l  /dev/md0

Disk /dev/md0: 931.4 GiB, 1000068874240 bytes, 1953259520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x9c0ff432

Device     Boot    Start        End    Sectors   Size Id Type
/dev/md0p1          2040       2055         16     8K 82 Linux swap / Solaris
/dev/md0p2 *     7809024   66398207   58589184    28G 83 Linux
/dev/md0p3      66398208 1953253375 1886855168 899.7G 83 Linux

Someone suggests that grub only works on an inode size of 128

# tune2fs -l /dev/md0p2 | grep -i 'inode size'
Inode size:               256

I don't see any reason the inode size would have changed, so I'm not sure I should care.

I'm stuck.

The partition table indicates a strange swap size. Maybe I screwed up this partition, the detection from Testdisk was incorrect and it wrote the wrong table we're seeing here. Anyway, it is not that bad. I guess I can always change this when I need.

gparted shows the following:

Partition     File System   Mount Point   Size         Flags
/dev/md0p1                                8 KiB
unallocated   unallocated                 3.72 GiB
/dev/md0p2    ext4          /mnt          27.94 GiB    boot
/dev/md0p3    ext4                        899.72 GiB
unallocated   unallocated                 3.72 GiB

Looks like somehow the end of my /home partition (/dev/md0p3) was cut.

No mention to the LVM.

Should I recreate /dev/md0p1 as swap adding the unallocated space close to it (and forget about the 4 GB lost at the end), or would playing with gparted here only make things worse?

Using a LVM is not of great interest in this case. The 1 TB disk allows me to reserve a comfortable 30 GB to the system, only 5 GB of which is used. I don't mind losing the LVM in the process unless I end up with a broken setup.

Edit 11: Data saved, giving up on the filesystem

At this point I was able to mount /home and etc and rsync that on another disk keeping permissions and all, so reinstalling from scratch is not really an issue.

I'd be happy if I understood what happened but losing hours fixing the LVM to end up with a setup I wouldn't fully understand and trust is not a great move, so I decided to reinstall from scratch.

I could reinstall on the array with the Debian Installer but I would get the same error at boot. I had to struggle with the Installer to destroy and recreate the array and ultimately, all went fine.

Another lesson learnt. From now on, I'll make even more backups. To backup /home, I rsync it on another disk. To save /etc and package list, here is what I do at work:

I use etckeeper to version  /etc, then I clone it onto another drive. And I use apt-clone to keep a list of installed packages.

#!/bin/sh
#
# Script to clone installation
#

DEST_DIR=/mnt/backup_drive

# Apt clone
apt-clone clone $DEST_DIR/apt-clone/apt-clone-$(lsb_release -si)-$(lsb_release -sc)-$(lsb_release -sr)-$(date +%F).tar.gz

# Git clone /etc
cd $DEST_DIR/etc; git pull --quiet

Answer 1

Your information is contradictory. /dev/md0 cannot simultaneously be a PV and have a partition table. file would recognize a PV. It seems that md0 is partitioned so that the LVM volume is rather /dev/md0p1 or /dev/md0p2.

Maybe for some reason pvscan / vgscan ignore /dev/md0 / /dev/md0p1 (and thus LVM cannot find the UUID). You may run pvscan through strace in order to check which block devices are scanned:

strace -e trace=open pvscan 2>&1 | grep /dev/md