Can I enable cephx authentication using a pool in qemu/kvm?

Question

(CentOS 7, Ceph Incunabulus, everything patched up to date)

Background

I've got a working converged Ceph/qemu/kvm cluster with cephx enabled for authentication. I'm trying to use this to store block devices for qemu/kvm virtual machines - not an uncommon configuration. I'd like to set up a pool for virtual disks in ceph, define that pool and authenticate to it in libvirt, and create virtual machines using the pool. All the tutorials and examples seem to show setting up cephx authentication directly on each disk file. Given my day so far, I'm beginning to see why!

Question

When I try the below, qemu-kvm appears to get passed a drive configured as file=rbd:vmlive/testguest-vda:auth_supported=none. That should read auth_supported=cephx, and for some reason it ain't happening!

Can I put all the authentication in the pool as I want, or do I have to put an authentication section in each virtual disk's XML? If I can do this, where's my mistake in what's below?

What I'm trying now

Context

I have two nodes: test1 and test2. Each has two OSDs, each is also a monitor (yes, I'm aware running with an even number of monitors is a bad idea). Each has libvirt installed, hence qemu/kvm. I'm running the commands below as root on test1. The UUID has been changed, and you'll see I don't make any keys visible below!

Create a Ceph pool and a client.kvm user

SENSIBLE_PGS=512 # Calculate as OSDs * 100 / number of replicas, rounded up to next power of 2
ID=kvm
POOL=vmlive
ceph osd pool create $POOL $SENSIBLE_PGS $SENSIBLE_PGS
KEY=`ceph auth get-or-create client.$ID | grep -v '^\[' | awk '{print $3}'`
ceph auth caps client.$ID mon "allow r" osd "allow rwx pool=$POOL"

Create a libvirt secret holding the Ceph user's key

SECRET_DEFINITION_FILE=~/secret-$ID.xml
SECRET_UUID=12345678-1234-1234-1234-123456789abc
cat > $SECRET_DEFINITION_FILE << EOF
<secret ephemeral='no' private='no'>
  <uuid>$SECRET_UUID</uuid>
  <usage type='ceph'>
    <name>ceph client.$ID key</name>
  </usage>
</secret>
EOF
virsh secret-define --file $SECRET_DEFINITION_FILE
rm -f $SECRET_DEFINITION_FILE
virsh secret-set-value "$SECRET_UUID" "$KEY"

Create a libvirt storage pool, trying to use cephx authentication

POOL_DEFINITION_FILE=~/pool-$POOL.xml
cat > $POOL_DEFINITION_FILE << EOF
<pool type="rbd">
  <name>$POOL</name>
  <source>
    <name>$POOL</name>
    <host name="test1" port="6789" />
    <host name="test2" port="6789" />
    <auth username='$ID' type='ceph'>
      <secret uuid='$SECRET_UUID'/>
    </auth>
  </source>
</pool>
EOF
virsh pool-define $POOL_DEFINITION_FILE
rm -f $POOL_DEFINITION_FILE
virsh pool-autostart $POOL
virsh pool-start $POOL

Create a guest using virt-install

NAME=testguest
DEV=vda
SIZE=8G
IMAGE=CentOS-7-x86_64-Minimal-1503-01.iso
VCPUS=1
RAM=512
MACLAST_HEX=10
IPLAST_DECIMAL=16

FILE=$NAME-$DEV
qemu-img create -f rbd rbd:$POOL/$FILE $SIZE
sudo virsh net-update default add-last ip-dhcp-host \
  --xml "<host mac='52:54:00:00:00:$MACLAST_HEX' ip='192.168.122.$IPLAST_DECIMAL' name='$NAME'/>" --live --config
sudo virt-install \
  --connect qemu:///system \
  --virt-type kvm \
  --name $NAME \
  --ram $RAM \
  --vcpus=$VCPUS \
  --disk vol=$POOL/$FILE \
  --location /var/lib/libvirt/images/$IMAGE \
  --vnc \
  --noautoconsole \
  --os-type linux \
  --os-variant rhel7 \
  --network=bridge:virbr0,model=virtio,mac=52:54:00:00:00:$MACLAST_HEX \
  --autostart

The result

WARNING  vmlive/testguest-vda may not be accessible by the hypervisor. You will need to grant the 'qemu' user search permissions for the following directories: ['vmlive', '']

Starting install...
Retrieving file .treeinfo...
Retrieving file vmlinuz...
Retrieving file initrd.img...
ERROR    internal error: process exited while connecting to monitor: 2016-01-06T15:16:54.639890Z qemu-kvm: -drive file=rbd:vmlive/testguest-vda:auth_supported=none:mon_host=test1\:6789,if=none,id=drive-virtio-disk0,format=raw: error connecting
2016-01-06T15:16:54.640574Z qemu-kvm: -drive file=rbd:vmlive/testguest-vda:auth_supported=none:mon_host=test1\:6789,if=none,id=drive-virtio-disk0,format=raw: could not open disk image rbd:vmlive/testguest-vda:auth_supported=none:mon_host=test1\:6789: Could not open 'rbd:vmlive/testguest-vda:auth_supported=none:mon_host=test1\:6789': Operation not supported

Domain installation does not appear to have been successful.

If you've read this far... thanks, congratulations, and all help gratefully received!

Update 1

Update: This appears to be a problem with virt-install partially (but incompletely) parsing pools. If I use --print-xml the partial XML generated for the disk is:

<disk type="network" device="disk">
  <driver name="qemu"/>
  <source protocol="rbd" name="vmlive/testguest-vda">
    <host name="test1" port="6789"/>
  </source>
  <target dev="vda" bus="virtio"/>
</disk>

... which includes much of the pool definition, but not the auth stanza.

I'll keep investigating.

HalosGhost · Accepted Answer · 2016-11-15 17:39:09Z

As you say, the XML not include auth stanza, that's why domain installation faild, but you can add the auth part manual
Edit /usr/share/virt-manager/virtinst/guest.py as follow

#vim /usr/share/virt-manager/virtinst/guest.py
import re
...
#define the auth 
auth_secret = '''
      <auth username='libvirt'>
        <secret type='ceph' uuid='e63e4b32-280e-4b00-982a-9d3xxxxxxx'/>
      </auth>
'''
ceph_monitors = '''
        <host name='172.16.200.104' port='6789'/>
        <host name='172.16.200.105' port='6789'/>
        <host name='172.16.200.106' port='6789'/>
'''

#change func: _build_xml 
    def _build_xml(self, is_initial):
        log_label = is_initial and "install" or "continue"
        disk_boot = not is_initial

        start_xml = self._get_install_xml(install=True, disk_boot=disk_boot)
        final_xml = self._get_install_xml(install=False)

#add------------start
        rgx_qemu = re.compile('(<driver name="qemu"[^>]*?>)')
        rgx_auth = re.compile('(?<=<source protocol="rbd" name=")([^>]*?">).*?(?= *?</source>)',re.S)

        start_xml = rgx_qemu.sub('\\1' + auth_secret,start_xml)
        start_xml = rgx_auth.sub('\\1' + ceph_monitors,start_xml)

        final_xml = rgx_qemu.sub('\\1' + auth_secret,final_xml)
        final_xml = rgx_auth.sub('\\1' + ceph_monitors,final_xml)
#add------------end

        logging.debug("Generated %s XML: %s",
                      log_label,
                      (start_xml and ("\n" + start_xml) or "None required"))
        logging.debug("Generated boot XML: \n%s", final_xml)

        return start_xml, final_xml

Then, run virt-install again

sudo virt-install \
  --connect qemu:///system \
  --virt-type kvm \
  --name $NAME \
  --ram $RAM \
  --vcpus=$VCPUS \
  --disk vol=$POOL/$FILE \
  --location /var/lib/libvirt/images/$IMAGE \
  --vnc \
  --noautoconsole \
  --os-type linux \
  --os-variant rhel7 \
  --network=bridge:virbr0,model=virtio,mac=52:54:00:00:00:$MACLAST_HEX \
  --autostart

More info http://www.isjian.com/ceph/virt-install-create-vm-use-rbd-pool/

XANi · Accepted Answer · 2020-11-26 09:26:58Z

Current method to do it (still broken as of virt-inst 3.1):

Setup secrets as for any other method
Create storage pool with rbd. XML will look something like this:

 <pool type='rbd'>
 <name>volumes</name> <!-- libvirt pool name -->
 <source>
  <host name='10.1.1.1' port='6789'/>
  <host name='10.1.1.2' port='6789'/>
  <host name='10.1.1.3' port='6789'/>
  <name>volumes</name><!-- ceph pool name -->
  <auth type='ceph' username='hypervisor'>
    <secret uuid='ff831b04-65a6-4b1b-ae53-9318e65158e6'/>
  </auth>
</source>
</pool>

Probably want to keep libvirt and ceph pool names the same for simplicity

add it (copying XML to /etc/libvirt/storage and making a link in ./autostart dir works too if you want to automate it, just needs libvirtd restart)

virsh pool-define volumes.xml 
virsh pool-start volumes

From that point you can just pass vol=poolname/volumename to the --disk option of the virt-install to specify it.

Peter Crowther · Accepted Answer · 2016-01-18 09:24:02Z

0

This is presently a known bug in virt-manager - see the email thread at https://www.redhat.com/archives/virt-tools-list/2016-January/msg00007.html.

In the meantime, I'll find another way round the problem.

answered Jan 18, 2016 at 9:24

Peter Crowther

416 bronze badges

Add a comment |

Stack Exchange Network

Can I enable cephx authentication using a pool in qemu/kvm?

Background

Question

What I'm trying now

Context

Create a Ceph pool and a client.kvm user

Create a libvirt secret holding the Ceph user's key

Create a libvirt storage pool, trying to use cephx authentication

Create a guest using virt-install

The result

Update 1

3 Answers 3

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
centos
kvm
qemu
ceph
.

Hot Network Questions

Can I enable cephx authentication using a pool in qemu/kvm?

Background

Question

What I'm trying now

Context

Create a Ceph pool and a client.kvm user

Create a libvirt secret holding the Ceph user's key

Create a libvirt storage pool, trying to use cephx authentication

Create a guest using virt-install

The result

Update 1

3 Answers 3

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged centoskvmqemuceph.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
centos
kvm
qemu
ceph
.