A story fairly similar to this one happened to me yesterday.
Update:
My only hard drive /dev/sda was originally partitioned this way:
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 1026047 512000 83 Linux
/dev/sda2 1026048 976773119 487873536 83 Linux
/dev/sda1 is the /boot partition, it is not encrypted. The /dev/sda2 partition was originally a LUKS partition containing the volume group fedora_pedro, which itself contained the following logical volumes:
/dev/mapper/fedora_pedro-root
/dev/mapper/fedora_pedro-home
/dev/mapper/fedora_pedro-swap
I was on a live disk, just about to shrink the /dev/sda2 partition following those steps when I abruptly had to stop and unplug my computer. I just had mounted the partition using
cryptsetup luksOpen /dev/sda2 fedora_pedro
and had a look around the place (first time I was playing with LVM & LUKS), mounting and unmounting the different logical volumes (swap, root, home), when I had to hastily shut the terminal down without closing the LUKS partition (exit; exit). Not really paying attention, I withdrew the live CD before the end of the shutdown procedure and turned the laptop off pressing the power button.
When I started the computer a few hours later, the boot hung at "Reached Target Basic System"; the only hint I got from booting from the hard drive in rescue mode was
dracut-initqueue [...] unit file of systemd-cryptsetup@luksMYUUID changed on disk
Using a rescue disk, I opened the drive (cryptsetup luksOpen /dev/sda2 fedora_pedro) and I inquired the status of the partition using e2fsck, which returned a pretty
fsck.ext4: Superblock invalid, trying backup blocks...
fsck.ext4: Bad magic number in super-block while trying to open /dev/sda2
pvdisplay returned that all PE were free (so the volume was shown as virtually empty), and vgscan shared that opinion, too, mentioning that my volume group did not contain any logical volume. lvdisplay returned nothing.
I had not resized anything yet, so I was very puzzled by the problem.
testdisk /dev/mapper/fedora_pedro
showed the right volumes (swap, root and home) and allowed access to all data on the system, so I chose to Write partition to the disk, hoping that I could see them after rebooting. That did not happen however, although fdisk -l /dev/mapper/fedora_pedro now returns
Disk /dev/mapper/fedora_pedro: 465.3 GiB, 49958403712 bytes, 975742976 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 / 4096 bytes
Disklabel type: dos
Disk identifier: 0x0000000
Device Boot Start End Blocks Id System
/dev/mapper/fedora_pedro1 * 2048 8161279 4079616 82 Linux swap / Solaris
/dev/mapper/fedora_pedro2 8161280 870885375 431362048 82 Linux
/dev/mapper/fedora_pedro3 870885376 975742975 52428800 82 Linux
I'm still not able to mount the volumes individually as I was able to unmount them when everything was O.K.. In fact, I cannot find them in /dev/mapper.
Additionally, since I copied the table to the disk, that I cannot see the physical volumes anymore, let alone the groups and the logical volumes (pvscan, vgscan & lvscan commands return nothing). But I remain confident and I am sure that the solution to this problem is not so far fetched:
- Maybe renaming the volume entries in the partition table to match the original ones in the grub. <-- Unlikely
Maybe find a backup and restore the system usingSee belowvgcfgrestoreor equivalent. <-- Possible- Maybe rewrite the superblocks on the disk using
mke2fs -S, as was proposed in the story I mentioned at the beginning.testdiskis positive about the size of the blocks: 4096. <-- Risky
But all those solutions (and the other ones you might propose) go beyond my limited knowledge of the logical volume system, as the reading may have disclosed already. I would like to know which procedure would be suitable to boot again properly.
News from the front
Again, I accessed my old system using testdisk. As mentioned before, the partitions are fine and I had a peek at my old file system, especially /etc to see if I could fetch some valuable information to restore. The longer this game goes on, the more I have the feeling that it is just a matter of labeling. I copied /etc/lvm to the live Documents file and tried to use it as a backup file for vgcfgrestore, but it returned a poor
Couldn't find device with uuid cZ83jX-WXkk-tNG4-ulGT-sAqq-HlKq-Omtqc8.
PV unknown device missing from cache
Format-specific setup for unknown device failed
Restore failed.
In fact, blkid does not return any uuid similar to that. What is more thrilling (and probably my own deed), is the fact that blkid returns this for my Ext4 file system:
/dev/sda1: "9f2a5417-dce6-45c1-8d93-1ee753e5c75c" TYPE="ext4" PARTUUID="0006282d-01"
/dev/sda2: "a33beb04-1147-446e-8f3b-7ff9d6bc226b" TYPE="crypto_LUKS" PARTUUID="0006282d-02"
[...]
/dev/mapper/fedora_pedro: PTTYPE="dos"
I think I made a mistake with Write partition to disk in testdisk, as the uuid of my current fedora_pedro does in no way match with the one of the original one. In fact, fedora_pedro has no uuid. Is it possible to attribute to this volume all the specifications of the previous, working unit: uuid, logical partitioning, ... If yes, how?
Getting closer?
vgchange -ay fedora_pedro returns
Volume group "fedora_pedro" not found
Skipping volume group fedora_pedro
Even though I use the same volume group name as the original one, mentioned in /etc/lvm/backup/ or /etc/lvm/archive. vgdisplay says exactly the same.
What's interesting is the output of lvmdiskscan
/dev/mapper/live-rw [ 4.00 GiB]
/dev/sda1 [ 500.00 MiB]
/dev/mapper/live-base [ 4.00 GiB]
/dev/loop2 [ 886.75 MiB]
/dev/sda2 [ 465.27 GiB]
/dev/mapper/live-osimg-min [ 4.00 GiB]
/dev/mapper/loop3 [ 4.00 GiB]
/dev/mapper/fedora_pedro [ 465.27 GiB] LVM physical volume
/dev/loop4 [ 512.00 MiB]
3 disks
5 partitions
1 LVM physical volume whole disk
0 LVM physical volume
If I understand it correctly, someone knows that there is a physical volume out there. But it does not want to show it.
$ pvdisplay fedora_pedro
Failed to read physical volume "fedora_pedro"
$ pvdisplay /dev/mapper/fedora_pedro
No physical volume found in lvmetad cache for /dev/mapper/fedora_pedro
Failed to read physical volume "/dev/mapper/fedora_pedro"
$ pvdisplay /home/liveuser/Documents/etc/lvm/archive/fedora_pedro_00000-1031984471.vg
Failed to read physical volume "/home/liveuser/Documents/etc/lvm/archive/fedora_pedro_00000-1031984471.vg"
I'm not sure how to go further. I did lvmdump -m and had a look at the dmsetup_ls_tree, dmsetup_table, dmsetup_status and dmsetup_info. dmsetup_info returns the following UUID for fedora_pedro
Name Maj Min Stat Open Targ Event UUID
fedora_pedro 253 3 L--w 0 1 0 CRYPT-LUKS1-a33beb04-1147-446e-8f3b-7ff9d6bc226b-fedora_pedro
This is the same UUID as the one returned by blkid for /dev/sda2. It is also the same as what I find at line 26 of /home/liveuser/Documents/etc/lvm/archive/fedora_pedro_00000-1031984471.vg
[...]
pv0 {
id = "cZ83jX-WXkk-tNG4-ulGT-sAqq-HlKq-Omtqc8"
device = "/dev/mapper/luks-a33beb04-1147-446e-8f3b-7ff9d6bc226b" # Hint only
[...]
and in the grub.cfg file on /dev/sda1.
As I wrote in the list above, will try to match the id's and UUID together, but I think the glitch is somewhere else.
vgchange -ay fedora_hostname. After that what is the output ofvgdisplay -v fedora_hostname?sda2UUID is reused in DM device names. Probably means: "This is the decrypted version of $UUID". That it is detected as PV but cannot be scanned may mean that the LVM metadata has been destroyed. If the LVs have been created without fragmentation then it should be possible to recreate them (with the correct size) to get access to the data. I am not familiar withtestdiskbut it should show you where the LVs are. You need to know their size and must (re)create them in the right order. IMHO this cannot cause (more) damage.