A few days ago our Ceph Cluster died - nobody knows why, but it seems that the monitors just got deleted. When trying to re-add them to the ceph.conf we get timeouts when starting ceph and these errors:
7f1b04382700 0 -- :/3908692206 >> 172.16.50.2:6789/0 pipe(0x7f1b0005a060 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f1b0005e350).fault
(172.16.50.2 is a node where a ceph monitor should be running...)
Since we were not able to restart the cluster (Proxmox) we wanted to extract the images from all our VMs from the Ceph OSDs. To do so I wrote a script collecting information about the data from the OSDs and reassemble the collected files.
The files have names like
Where 147082238e1f29 seems to be the ID of the VM-Image and 0000000000009238 seems to be the "offset". So I listed all files with the same ID and sorted them respecting their offset. The problem is: There are gaps... Not all offsets are present on the OSDs.
So when putting all these files together with dd, the image seems to be corrupted - because there are gaps.
Can anyone help us with the timeouts when starting ceph or at least help us with extracting the VM images?