How to configure AppArmor so that KVM can start guest that has a backing file chain

Question

I'm playing around with using Qemu external snapshots for KVM (because they are so fast!) on Ubuntu 16.04 and have the following setup:

• All files for the VM are within a single directory (named after the guest).
• There is a head.qcow2 symlink that always points to the top "head" qcow2 image. This way I don't have to keep updating the VM config. (virsh edit).
• The original/bottom image is called base.qcow2

I use the following script to take a snapshot (create a new qcow2 image and make it the head). The script is only executed when the guest is not running.

#!/bin/bash
UNIX_TIMESTAMP=$(date +%s)
CURRENT_BACKING_FILE=`readlink head.qcow2`
NEW_HEAD_FILE="`pwd`/`echo $UNIX_TIMESTAMP`.qcow2"

# Create the new "head" disk image where all future changes are made.
sudo qemu-img create \
  -f qcow2 \
  -b $CURRENT_BACKING_FILE \
  $NEW_HEAD_FILE

# Update the symlink
rm head.qcow2
ln -s $NEW_HEAD_FILE head.qcow2

This will work fine when head.qcow2 points to base.qcow2, or I run the script once so there is only a single backing file. However, If there is a chain of backing files (e.g. a backing file also has a backing file), then I will get the following error:

error: Failed to start domain template-ubuntu-docker2
error: internal error: early end of file from monitor, possible problem: 2018-04-05T19:04:55.597373Z qemu-system-x86_64: -drive file=/media/ssd_storage2/kvm/vms/templates/template-ubuntu-docker2/head.qcow2,format=qcow2,if=none,id=drive-virtio-disk0: Could not open backing file: Could not open backing file: Could not open '/media/ssd_storage2/kvm/vms/templates/template-ubuntu-docker2/base.qcow2': Permission denied

The output of qemu-img --backing-chain head.qcow2 is:

image: head.qcow2
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 196K
cluster_size: 65536
backing file: /media/ssd_storage2/kvm/vms/templates/template-ubuntu-docker2/1522954330.qcow2
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

image: /media/ssd_storage2/kvm/vms/templates/template-ubuntu-docker2/1522954330.qcow2
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 4.1M
cluster_size: 65536
backing file: base.qcow2 (actual path: /media/ssd_storage2/kvm/vms/templates/template-ubuntu-docker2/base.qcow2)
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

image: /media/ssd_storage2/kvm/vms/templates/template-ubuntu-docker2/base.qcow2
file format: qcow2
virtual size: 20G (21474836480 bytes)
disk size: 8.8G
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: true
    refcount bits: 16
    corrupt: false

I've played around with deleting images or using qemu-img commit to merge them, and as soon as I go down to just one or two images the guest will start.

Since it looked like a permission error with "could not open" I tried setting permissions to 777 and I still get the same error message, so it shouldn't be a permissions issue.

What do I need to do?

Answer 1

It looks like it's related to AppArmor again (I had a similar issue some years ago on Ubuntu 14.04 that I forgot about). I was able to get it working by:

Installing apparmor-utils

sudo apt-get install apparmor-utils

Then find the libvirt ID in the most recent syslogs:

sudo cat /var/log/syslog | grep "apparmor" | grep "DENIED" | grep libvirt-

Then use that ID with the full path to set it to "complain mode".

sudo aa-complain \
/etc/apparmor.d/libvirt/libvirt-b0cd5bdf-6dfc-4f75-bc0f-e38adff1cdd2

Then start your guest.

This feels like a workaround at best and annoying if I have to do it for each guest individually. I would be happier if someone provided a better solution.

Answer 2

Got same problem on Debian Buster, the fact is that libvirt apparmor profiles are dynamically created and stored under /etc/apparmor.d/libvirt/ (thanks @Programster). In this directory you should found a TEMPLATE.qemu similar to this one:

#
# This profile is for the domain whose UUID matches this file.
#

#include <tunables/global>

profile LIBVIRT_TEMPLATE flags=(attach_disconnected) {
  #include <abstractions/libvirt-qemu>
}

You should add rules for the directory where you store your qcow2 in the profile:

#
# This profile is for the domain whose UUID matches this file.
#

#include <tunables/global>

profile LIBVIRT_TEMPLATE flags=(attach_disconnected) {
  #include <abstractions/libvirt-qemu>
  /media/ssd_storage2/kvm/vms/templates/template-ubuntu-docker2/**.qcow2 rk,
}

"r" is for read permission and "k" is for lock permissions.

You can track apparmor audit messages with "dmesg -w", and get similar messages depending which permissions are missing:

[ 4404.140981] audit: type=1400 audit(1566342100.488:254): apparmor="DENIED" operation="open" profile="libvirt-b24e7663-d71a-4970-9d6c-711d43629135" name="/home/libvirt/0.0.6-dev.19+20~811d1dc983b5.qcow2" pid=22384 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=64055 ouid=64055
...
[ 4652.827860] audit: type=1400 audit(1566342349.172:263): apparmor="DENIED" operation="file_lock" profile="libvirt-b24e7663-d71a-4970-9d6c-711d43629135" name="/home/libvirt/0.0.6-dev.19+20~811d1dc983b5.qcow2" pid=23085 comm="qemu-system-x86" requested_mask="k" denied_mask="k" fsuid=64055 ouid=64055