A story fairly similar to this one happened to me yesterday.

Update: My only hard drive /dev/sda was originally partitioned this way:

Device     Boot   Start        End     Blocks  Id  System
/dev/sda1  *       2048    1026047     512000  83   Linux
/dev/sda2       1026048  976773119  487873536  83   Linux

/dev/sda1 is the /boot partition, it is not encrypted. The /dev/sda2 partition was originally a LUKS partition containing the volume group fedora_pedro, which itself contained the following logical volumes:


I was on a live disk, just about to shrink the /dev/sda2 partition following those steps when I abruptly had to stop and unplug my computer. I just had mounted the partition using

cryptsetup luksOpen /dev/sda2 fedora_pedro

and had a look around the place (first time I was playing with LVM & LUKS), mounting and unmounting the different logical volumes (swap, root, home), when I had to hastily shut the terminal down without closing the LUKS partition (exit; exit). Not really paying attention, I withdrew the live CD before the end of the shutdown procedure and turned the laptop off pressing the power button.

When I started the computer a few hours later, the boot hung at "Reached Target Basic System"; the only hint I got from booting from the hard drive in rescue mode was

dracut-initqueue [...] unit file of systemd-cryptsetup@luksMYUUID changed on disk

Using a rescue disk, I opened the drive (cryptsetup luksOpen /dev/sda2 fedora_pedro) and I inquired the status of the partition using e2fsck, which returned a pretty

fsck.ext4: Superblock invalid, trying backup blocks...
fsck.ext4: Bad magic number in super-block while trying to open /dev/sda2

pvdisplay returned that all PE were free (so the volume was shown as virtually empty), and vgscan shared that opinion, too, mentioning that my volume group did not contain any logical volume. lvdisplay returned nothing.

I had not resized anything yet, so I was very puzzled by the problem.

testdisk /dev/mapper/fedora_pedro

showed the right volumes (swap, root and home) and allowed access to all data on the system, so I chose to Write partition to the disk, hoping that I could see them after rebooting. That did not happen however, although fdisk -l /dev/mapper/fedora_pedro now returns

Disk /dev/mapper/fedora_pedro: 465.3 GiB, 49958403712 bytes, 975742976 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 / 4096 bytes
Disklabel type: dos
Disk identifier: 0x0000000

Device                      Boot     Start        End     Blocks  Id  System
/dev/mapper/fedora_pedro1  *         2048    8161279    4079616  82  Linux swap / Solaris
/dev/mapper/fedora_pedro2         8161280  870885375  431362048  82  Linux
/dev/mapper/fedora_pedro3       870885376  975742975   52428800  82  Linux

I'm still not able to mount the volumes individually as I was able to unmount them when everything was O.K.. In fact, I cannot find them in /dev/mapper. Additionally, since I copied the table to the disk, that I cannot see the physical volumes anymore, let alone the groups and the logical volumes (pvscan, vgscan & lvscan commands return nothing). But I remain confident and I am sure that the solution to this problem is not so far fetched:

  • Maybe renaming the volume entries in the partition table to match the original ones in the grub. <-- Unlikely
  • Maybe find a backup and restore the system using vgcfgrestore or equivalent. <-- Possible See below
  • Maybe rewrite the superblocks on the disk using mke2fs -S, as was proposed in the story I mentioned at the beginning. testdisk is positive about the size of the blocks: 4096. <-- Risky

But all those solutions (and the other ones you might propose) go beyond my limited knowledge of the logical volume system, as the reading may have disclosed already. I would like to know which procedure would be suitable to boot again properly.

News from the front

Again, I accessed my old system using testdisk. As mentioned before, the partitions are fine and I had a peek at my old file system, especially /etc to see if I could fetch some valuable information to restore. The longer this game goes on, the more I have the feeling that it is just a matter of labeling. I copied /etc/lvm to the live Documents file and tried to use it as a backup file for vgcfgrestore, but it returned a poor

Couldn't find device with uuid cZ83jX-WXkk-tNG4-ulGT-sAqq-HlKq-Omtqc8.
PV unknown device missing from cache
Format-specific setup for unknown device failed
Restore failed.

In fact, blkid does not return any uuid similar to that. What is more thrilling (and probably my own deed), is the fact that blkid returns this for my Ext4 file system:

/dev/sda1: "9f2a5417-dce6-45c1-8d93-1ee753e5c75c" TYPE="ext4" PARTUUID="0006282d-01"
/dev/sda2: "a33beb04-1147-446e-8f3b-7ff9d6bc226b" TYPE="crypto_LUKS" PARTUUID="0006282d-02"
/dev/mapper/fedora_pedro: PTTYPE="dos"

I think I made a mistake with Write partition to disk in testdisk, as the uuid of my current fedora_pedro does in no way match with the one of the original one. In fact, fedora_pedro has no uuid. Is it possible to attribute to this volume all the specifications of the previous, working unit: uuid, logical partitioning, ... If yes, how?

Getting closer?

vgchange -ay fedora_pedro returns

Volume group "fedora_pedro" not found
Skipping volume group fedora_pedro

Even though I use the same volume group name as the original one, mentioned in /etc/lvm/backup/ or /etc/lvm/archive. vgdisplay says exactly the same.

What's interesting is the output of lvmdiskscan

/dev/mapper/live-rw        [   4.00 GiB]
/dev/sda1                  [ 500.00 MiB]
/dev/mapper/live-base      [   4.00 GiB]
/dev/loop2                 [ 886.75 MiB]
/dev/sda2                  [ 465.27 GiB]
/dev/mapper/live-osimg-min [   4.00 GiB]
/dev/mapper/loop3          [   4.00 GiB]
/dev/mapper/fedora_pedro   [ 465.27 GiB] LVM physical volume
/dev/loop4                 [ 512.00 MiB]
3 disks
5 partitions
1 LVM physical volume whole disk
0 LVM physical volume

If I understand it correctly, someone knows that there is a physical volume out there. But it does not want to show it.

$ pvdisplay fedora_pedro
Failed to read physical volume "fedora_pedro"

$ pvdisplay /dev/mapper/fedora_pedro
No physical volume found in lvmetad cache for /dev/mapper/fedora_pedro
Failed to read physical volume "/dev/mapper/fedora_pedro"

$ pvdisplay /home/liveuser/Documents/etc/lvm/archive/fedora_pedro_00000-1031984471.vg
Failed to read physical volume "/home/liveuser/Documents/etc/lvm/archive/fedora_pedro_00000-1031984471.vg"

I'm not sure how to go further. I did lvmdump -m and had a look at the dmsetup_ls_tree, dmsetup_table, dmsetup_status and dmsetup_info. dmsetup_info returns the following UUID for fedora_pedro

Name         Maj Min Stat Open Targ Event UUID
fedora_pedro 253   3 L--w    0    1     0 CRYPT-LUKS1-a33beb04-1147-446e-8f3b-7ff9d6bc226b-fedora_pedro

This is the same UUID as the one returned by blkid for /dev/sda2. It is also the same as what I find at line 26 of /home/liveuser/Documents/etc/lvm/archive/fedora_pedro_00000-1031984471.vg

pv0 {
       id = "cZ83jX-WXkk-tNG4-ulGT-sAqq-HlKq-Omtqc8"
       device = "/dev/mapper/luks-a33beb04-1147-446e-8f3b-7ff9d6bc226b" # Hint only

and in the grub.cfg file on /dev/sda1.

As I wrote in the list above, will try to match the id's and UUID together, but I think the glitch is somewhere else.

  • After having read that several times I am still not sure what your disk looked like. What layer is encrypted, the whole disk, a single partition, an LV? What is the PV? "so I chose to copy the table to the disk" - Does that mean you had a partition table within a LUKS volume / disk and have copied that to the disk itself? That would have destroyed the LUKS header. – Hauke Laging Nov 22 '14 at 2:16
  • Run the command vgchange -ay fedora_hostname. After that what is the output of vgdisplay -v fedora_hostname? – Hauke Laging Nov 23 '14 at 5:54
  • I think it's not a problem that the sda2 UUID is reused in DM device names. Probably means: "This is the decrypted version of $UUID". That it is detected as PV but cannot be scanned may mean that the LVM metadata has been destroyed. If the LVs have been created without fragmentation then it should be possible to recreate them (with the correct size) to get access to the data. I am not familiar with testdisk but it should show you where the LVs are. You need to know their size and must (re)create them in the right order. IMHO this cannot cause (more) damage. – Hauke Laging Nov 24 '14 at 4:48

By some mystery I had wiped the volume group and the physical volume in it. It was just a matter of recovering them using the metadata I could back-up using testdisk. I copied the original /etc/lvm folder on the Desktop of the live user, then

$ pvcreate --uuid "cZ83jX-WXkk-tNG4-ulGT-sAqq-HlKq-Omtqc8" \
--restorefile /home/liveuser/Desktop/etc/lvm/archive/VG.vg /dev/sda2
  Physical volume "/dev/sda2" successfully created

$ vgcfgrestore VG
  Restored volume group VG

$ lvchange -ay /dev/VG/home /dev/VG/root /dev/VG/swap

$ lvs -a -o +devices

After that, vgdisplay -v fedora_pedro returned consistent output, mentioning the original logical and physical volumes, which I was allowed to mount and explore. A couple of minutes later, though, I unfortunately wiped the header of the luks container trying to write those changes to the disk. So all was done in vain...

  • 1
    As long as the system is running you can get the necessary data from dmsetup --showkeys table fedora_pedro. – Hauke Laging Nov 25 '14 at 1:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.