1

A few days ago our Ceph Cluster died - nobody knows why, but it seems that the monitors just got deleted. When trying to re-add them to the ceph.conf we get timeouts when starting ceph and these errors:

7f1b04382700  0 -- :/3908692206 >> 172.16.50.2:6789/0 pipe(0x7f1b0005a060 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f1b0005e350).fault

(172.16.50.2 is a node where a ceph monitor should be running...)

Since we were not able to restart the cluster (Proxmox) we wanted to extract the images from all our VMs from the Ceph OSDs. To do so I wrote a script collecting information about the data from the OSDs and reassemble the collected files.

The files have names like

current/1.382_head/DIR_2/DIR_8/DIR_3/rbd\udata.147082238e1f29.0000000000009238__head_CB25F382__1

Where 147082238e1f29 seems to be the ID of the VM-Image and 0000000000009238 seems to be the "offset". So I listed all files with the same ID and sorted them respecting their offset. The problem is: There are gaps... Not all offsets are present on the OSDs.

So when putting all these files together with dd, the image seems to be corrupted - because there are gaps.

Can anyone help us with the timeouts when starting ceph or at least help us with extracting the VM images?

Improve this question
3
  • I assume you've found about (ceph.com/planet/ceph-recover-a-rbd-image-from-a-dead-cluster) and that rbd commands are broken? Missing blocks isn't the end of the world, you should know rbd_restore would create an empty disk and then dd relevant blocks to the right places. What's missing may be legitimately blank, or could eventually be recovered through your enclosed filesystems' fsck. Patience, and good luck
    – SYN
    Dec 21, 2016 at 20:16
  • Yes, I found your link but the rbd commands do not work anymore (timeouts, see above...). So I tried using other tools I found at github. They all copy the files with dd and put them together via append. But what about the missing blocks? Maybe they are just empty and not saved for that reason. So I tried to find all missing blocks and replaced them with a 4MB blocks of zeroes - However, when repairing the partition with fsck I just get an empty root dir with a lost+found folder...
    – Kryptur
    Dec 21, 2016 at 23:13
  • Sorry to hear it. If you're sure you have gathered objects from all your OSDs prior to reconstructing your faulty disk, then it sounds pretty bad. In doubt: support on their IRC is usually pretty helpful in desperate situations (ceph.com/resources/mailing-list-irc)
    – SYN
    Dec 22, 2016 at 1:14

1 Answer 1

Reset to default
1

After some more prayers I actually got it:

I specified a wrong block size and I changed to copying with offsets instead of inserting "0-blocks".

To extract images from OSDs only (nothing else working):
https://gitlab.lbader.de/kryptur/ceph-recovery/tree/master

Now I am able to extract the VMs from the OSDs and copy the images to a fresh cluster.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .