I have a server that was set up using LVM. It ran for years just fine with out issue until about 10 days ago. When I tried to log into it, the server had crashed. When we look at the server using a KUBUNTU Live CD, "fdisk -l" shows that the hard drive still contains it's two paritions - (/dev/sdc1 with ext3 filesystem for the /boot partition and /dev/sdc2 which is set up with LVM). If I do a pvdisplay, LVM recognized /dev/sda2, and this LVM only had this one physical volume as it's member. If I do a vgdisplay the server sees no volume groups, and if I do an lvdisplay, it sees no logical volumes either. lvmdiskscan shows the pv on /dev/sda2 as it's supposed to. However vgscan finds nothing, and lvscan finds nothing. For safe keeping, I have made dd image backups of the entire hard drive so any failed attempts to fix this issue can be rolled back.
Obviously something has happened to the Volume Group metadata. I have read all over the web looking for solutions and have found very little that seems promising for out case, and there was no backup taken of this LVM structure, neither of the data beneath it. One thing I did notice in my search, is that a lot of people seem to have lost their Volume Group metadata after having attempted to resize their LVM. We did not manually attempt any resizing, but it's possible that the a Logical Volume could have grown to exceed the size of the volume group, and perhaps LVM could have automatically attempted to resize it's self, causing this loss of metadata. I can not verify this however, because all of the log files are down inside of one of the logical volumes beneath this Volume Group.
I did see one scenario that resembles this scenario, and where a repair was successful. Take a look at http://blog.adamsbros.org/2009/05/30/recover-lvm-volume-groups-and-logical-volumes-without-backups/ where someone was able to determine the offset values using pvck, and then drilling down into the partition with a hex editor, retrieving the values of the offsets, diffing two of the offsets, and creating a restore file using this info, and then recreating the LVM to restore the ability to find the data.
Does anyone here have any other ideas that might help in repairing this issue? I am willing to try to the example above, but I want the input of anyone here that might have successful experience in fixing the issue that we are facing.