1

I wanted to shrink the file-system and unfortunately I have missed one step:

Without resizing the file-system before, I reduced the logical volume.

The result was that the system is not booting any more and giving me a message that the file-system is corrupt.

[root@node2 ~]# mount -a
   mount: wrong fs type, bad option, bad superblock on /dev/mapper/vgprod-prod,
   missing codepage or helper program, or other error

   In some cases useful info is found in syslog - try
   dmesg | tail or so.

[root@node2 ~]# e2fsck -f /dev/mapper/vgprod-prod
   e2fsck 1.42.9 (28-Dec-2013)
   Error reading block 65536 (Invalid argument).  Ignore error<y>? yes
   Force rewrite<y>? yes
   Superblock has an invalid journal (inode 8).
   Clear<y>? yes
   *** ext3 journal has been deleted - filesystem is now ext2 only ***

   Superblock has_journal flag is clear, but a journal inode is present.
   Clear<y>? yes
   The filesystem size (according to the superblock) is 115712 blocks
   The physical size of the device is 64512 blocks
   Either the superblock or the partition table is likely to be corrupt!
   Abort<y>? yes
   Error writing block 65536 (Invalid argument).  Ignore error<y>? yes

   /dev/mapper/vgprod-prod: ***** FILE SYSTEM WAS MODIFIED *****

Please advise me how to fix the issues.

1
  • Please clarify whether you are still in the initramfs prompt, and/or have rebooted, and whether this is your root file-system block device. Dec 4, 2016 at 14:34

1 Answer 1

2

Before doing any more changes which might cause a lot of damage, make a sector-by-sector copy of the physical volume (as in: dd if=/dev/sdXY of=/path/to/safe/place.img) at its current state and store it somewhere unchanged. Use pvdisplay to know which partition to copy. If the worst happens (file system too damaged to understand its contents), you will still be able to use testdisk (if the filesystem resided on your LVM partition in one contiguous block):

  1. run testdisk /path/to/image.img
  2. select "Non partitioned media": testdisk doesn't know how to write LVM, but will still read information from the partitions it finds
  3. use "Analyze" to search for your partition; use "Deeper search" if your filesystem is not found
  4. Use P to list files, then :/a to select and c/C to copy your files.

Assuming that your failed LVM partition doesn't host your rootfs, back up your /etc/lvm directory, too. If the partition did host your rootfs, perhaps your previous backups contain this directory? We will need it. If all else fails, it might still be possible to find the contents of files from /etc/lvm/ using a hex editor and string search for description =, assuming that they are not fragmented, but that is a very long process.

If you are still in the initramfs shell and don't have enough free HDD space for the image, with luck (usb-storage and other modules present in initramfs) you may be able to mount a USB-HDD:

  1. plug the drive, wait a few seconds for it to spin up
  2. the kernel messages are likely to be shown to you, otherwise run: dmesg | tail
  3. look for lines which look as [ 8391.759613] sd 6:0:0:0: [sdX] 15826944 512-byte logical blocks: (8.10 GB/7.55 GiB) (compare the size to make sure that sdX is your USB-HDD) and [ 8391.770279] sdX: sdX1 (with sdX1 ... sdXn being the list of partitions)
  4. run:

    mkdir -p /mnt/usb-hdd # ensure existing mountpoint
    mount /dev/sdXN /mnt/usb-hdd # substitute X for the drive letter and N for the partition number from (3)
    
  5. write the backup image to and/or copy the restored /etc/lvm backup from /mnt/usb-hdd

  6. don't forget to unmount the thumbdrive after you are done: umount /mnt/usb-hdd

Having that done, take a look at your present /etc/lvm/backup and /etc/lvm/archive you fetched from the rootfs or from your backups. There might be backed up metadata created before you ran lvresize which caused all that damage. Look in the files inside these directories for a line beginning with description =. Try grep description .../etc/lvm/*/* to list descriptions; if you're still in the initramfs shell, use less, more, or, failing that, cat and Shift+PgUp/Shift+PgDn to view text files. Is there a file with Created *before* executing 'lvresize -l <something> /dev/vgprod/prod'? Run vgcfgrestore -f /etc/lvm/archive/<suitable archive file> vgprod to restore the metadata to their previous values and try to mount /dev/vgprod/prod afterwards.

In such sutuations you should avoid doing any kind of fsck before you have successfully copied your files and verified that they're safe or made an image of your partition which you are sure you are able to restore. fsck on a truncated filesystem is likely to make matters worse.

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .