(CentOS 7, Ceph Incunabulus, everything patched up to date)


I've got a working converged Ceph/qemu/kvm cluster with cephx enabled for authentication. I'm trying to use this to store block devices for qemu/kvm virtual machines - not an uncommon configuration. I'd like to set up a pool for virtual disks in ceph, define that pool and authenticate to it in libvirt, and create virtual machines using the pool. All the tutorials and examples seem to show setting up cephx authentication directly on each disk file. Given my day so far, I'm beginning to see why!


When I try the below, qemu-kvm appears to get passed a drive configured as file=rbd:vmlive/testguest-vda:auth_supported=none. That should read auth_supported=cephx, and for some reason it ain't happening!

Can I put all the authentication in the pool as I want, or do I have to put an authentication section in each virtual disk's XML? If I can do this, where's my mistake in what's below?

What I'm trying now


I have two nodes: test1 and test2. Each has two OSDs, each is also a monitor (yes, I'm aware running with an even number of monitors is a bad idea). Each has libvirt installed, hence qemu/kvm. I'm running the commands below as root on test1. The UUID has been changed, and you'll see I don't make any keys visible below!

Create a Ceph pool and a client.kvm user

SENSIBLE_PGS=512 # Calculate as OSDs * 100 / number of replicas, rounded up to next power of 2
ceph osd pool create $POOL $SENSIBLE_PGS $SENSIBLE_PGS
KEY=`ceph auth get-or-create client.$ID | grep -v '^\[' | awk '{print $3}'`
ceph auth caps client.$ID mon "allow r" osd "allow rwx pool=$POOL"

Create a libvirt secret holding the Ceph user's key

<secret ephemeral='no' private='no'>
  <usage type='ceph'>
    <name>ceph client.$ID key</name>
virsh secret-define --file $SECRET_DEFINITION_FILE
virsh secret-set-value "$SECRET_UUID" "$KEY"

Create a libvirt storage pool, trying to use cephx authentication

<pool type="rbd">
    <host name="test1" port="6789" />
    <host name="test2" port="6789" />
    <auth username='$ID' type='ceph'>
      <secret uuid='$SECRET_UUID'/>
virsh pool-define $POOL_DEFINITION_FILE
virsh pool-autostart $POOL
virsh pool-start $POOL

Create a guest using virt-install


qemu-img create -f rbd rbd:$POOL/$FILE $SIZE
sudo virsh net-update default add-last ip-dhcp-host \
  --xml "<host mac='52:54:00:00:00:$MACLAST_HEX' ip='192.168.122.$IPLAST_DECIMAL' name='$NAME'/>" --live --config
sudo virt-install \
  --connect qemu:///system \
  --virt-type kvm \
  --name $NAME \
  --ram $RAM \
  --vcpus=$VCPUS \
  --disk vol=$POOL/$FILE \
  --location /var/lib/libvirt/images/$IMAGE \
  --vnc \
  --noautoconsole \
  --os-type linux \
  --os-variant rhel7 \
  --network=bridge:virbr0,model=virtio,mac=52:54:00:00:00:$MACLAST_HEX \

The result

WARNING  vmlive/testguest-vda may not be accessible by the hypervisor. You will need to grant the 'qemu' user search permissions for the following directories: ['vmlive', '']

Starting install...
Retrieving file .treeinfo...
Retrieving file vmlinuz...
Retrieving file initrd.img...
ERROR    internal error: process exited while connecting to monitor: 2016-01-06T15:16:54.639890Z qemu-kvm: -drive file=rbd:vmlive/testguest-vda:auth_supported=none:mon_host=test1\:6789,if=none,id=drive-virtio-disk0,format=raw: error connecting
2016-01-06T15:16:54.640574Z qemu-kvm: -drive file=rbd:vmlive/testguest-vda:auth_supported=none:mon_host=test1\:6789,if=none,id=drive-virtio-disk0,format=raw: could not open disk image rbd:vmlive/testguest-vda:auth_supported=none:mon_host=test1\:6789: Could not open 'rbd:vmlive/testguest-vda:auth_supported=none:mon_host=test1\:6789': Operation not supported

Domain installation does not appear to have been successful.

If you've read this far... thanks, congratulations, and all help gratefully received!

Update 1

Update: This appears to be a problem with virt-install partially (but incompletely) parsing pools. If I use --print-xml the partial XML generated for the disk is:

<disk type="network" device="disk">
  <driver name="qemu"/>
  <source protocol="rbd" name="vmlive/testguest-vda">
    <host name="test1" port="6789"/>
  <target dev="vda" bus="virtio"/>

... which includes much of the pool definition, but not the auth stanza.

I'll keep investigating.


3 Answers 3


As you say, the XML not include auth stanza, that's why domain installation faild, but you can add the auth part manual
Edit /usr/share/virt-manager/virtinst/guest.py as follow

#vim /usr/share/virt-manager/virtinst/guest.py
import re
#define the auth 
auth_secret = '''
      <auth username='libvirt'>
        <secret type='ceph' uuid='e63e4b32-280e-4b00-982a-9d3xxxxxxx'/>
ceph_monitors = '''
        <host name='' port='6789'/>
        <host name='' port='6789'/>
        <host name='' port='6789'/>

#change func: _build_xml 
    def _build_xml(self, is_initial):
        log_label = is_initial and "install" or "continue"
        disk_boot = not is_initial

        start_xml = self._get_install_xml(install=True, disk_boot=disk_boot)
        final_xml = self._get_install_xml(install=False)

        rgx_qemu = re.compile('(<driver name="qemu"[^>]*?>)')
        rgx_auth = re.compile('(?<=<source protocol="rbd" name=")([^>]*?">).*?(?= *?</source>)',re.S)

        start_xml = rgx_qemu.sub('\\1' + auth_secret,start_xml)
        start_xml = rgx_auth.sub('\\1' + ceph_monitors,start_xml)

        final_xml = rgx_qemu.sub('\\1' + auth_secret,final_xml)
        final_xml = rgx_auth.sub('\\1' + ceph_monitors,final_xml)

        logging.debug("Generated %s XML: %s",
                      (start_xml and ("\n" + start_xml) or "None required"))
        logging.debug("Generated boot XML: \n%s", final_xml)

        return start_xml, final_xml

Then, run virt-install again

sudo virt-install \
  --connect qemu:///system \
  --virt-type kvm \
  --name $NAME \
  --ram $RAM \
  --vcpus=$VCPUS \
  --disk vol=$POOL/$FILE \
  --location /var/lib/libvirt/images/$IMAGE \
  --vnc \
  --noautoconsole \
  --os-type linux \
  --os-variant rhel7 \
  --network=bridge:virbr0,model=virtio,mac=52:54:00:00:00:$MACLAST_HEX \

More info http://www.isjian.com/ceph/virt-install-create-vm-use-rbd-pool/


Current method to do it (still broken as of virt-inst 3.1):

  1. Setup secrets as for any other method
  2. Create storage pool with rbd. XML will look something like this:
 <pool type='rbd'>
 <name>volumes</name> <!-- libvirt pool name -->
  <host name='' port='6789'/>
  <host name='' port='6789'/>
  <host name='' port='6789'/>
  <name>volumes</name><!-- ceph pool name -->
  <auth type='ceph' username='hypervisor'>
    <secret uuid='ff831b04-65a6-4b1b-ae53-9318e65158e6'/>

Probably want to keep libvirt and ceph pool names the same for simplicity

  1. add it (copying XML to /etc/libvirt/storage and making a link in ./autostart dir works too if you want to automate it, just needs libvirtd restart)
virsh pool-define volumes.xml 
virsh pool-start volumes
  1. From that point you can just pass vol=poolname/volumename to the --disk option of the virt-install to specify it.

This is presently a known bug in virt-manager - see the email thread at https://www.redhat.com/archives/virt-tools-list/2016-January/msg00007.html.

In the meantime, I'll find another way round the problem.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .