Before doing any more changes which might cause a lot of damage, make a sector-by-sector copy of the physical volume (as in: dd if=/dev/sdXY of=/path/to/safe/place.img
) at its current state and store it somewhere unchanged. Use pvdisplay
to know which partition to copy. If the worst happens (file system too damaged to understand its contents), you will still be able to use testdisk (if the filesystem resided on your LVM partition in one contiguous block):
- run
testdisk /path/to/image.img
- select "Non partitioned media":
testdisk
doesn't know how to write LVM, but will still read information from the partitions it finds
- use "Analyze" to search for your partition; use "Deeper search" if your filesystem is not found
- Use P to list files, then :/a to select and c/C to copy your files.
Assuming that your failed LVM partition doesn't host your rootfs, back up your /etc/lvm
directory, too. If the partition did host your rootfs, perhaps your previous backups contain this directory? We will need it. If all else fails, it might still be possible to find the contents of files from /etc/lvm/
using a hex editor and string search for description =
, assuming that they are not fragmented, but that is a very long process.
If you are still in the initramfs shell and don't have enough free HDD space for the image, with luck (usb-storage and other modules present in initramfs) you may be able to mount a USB-HDD:
- plug the drive, wait a few seconds for it to spin up
- the kernel messages are likely to be shown to you, otherwise run:
dmesg | tail
- look for lines which look as
[ 8391.759613] sd 6:0:0:0: [sdX] 15826944 512-byte logical blocks: (8.10 GB/7.55 GiB)
(compare the size to make sure that sdX
is your USB-HDD) and [ 8391.770279] sdX: sdX1
(with sdX1 ... sdXn being the list of partitions)
run:
mkdir -p /mnt/usb-hdd # ensure existing mountpoint
mount /dev/sdXN /mnt/usb-hdd # substitute X for the drive letter and N for the partition number from (3)
write the backup image to and/or copy the restored /etc/lvm
backup from /mnt/usb-hdd
- don't forget to unmount the thumbdrive after you are done:
umount /mnt/usb-hdd
Having that done, take a look at your present /etc/lvm/backup
and /etc/lvm/archive
you fetched from the rootfs or from your backups. There might be backed up metadata created before you ran lvresize
which caused all that damage. Look in the files inside these directories for a line beginning with description =
. Try grep description .../etc/lvm/*/*
to list descriptions; if you're still in the initramfs shell, use less
, more
, or, failing that, cat
and Shift+PgUp/Shift+PgDn to view text files. Is there a file with Created *before* executing 'lvresize -l <something> /dev/vgprod/prod'
? Run vgcfgrestore -f /etc/lvm/archive/<suitable archive file> vgprod
to restore the metadata to their previous values and try to mount /dev/vgprod/prod
afterwards.
In such sutuations you should avoid doing any kind of fsck
before you have successfully copied your files and verified that they're safe or made an image of your partition which you are sure you are able to restore. fsck
on a truncated filesystem is likely to make matters worse.