SELinux blocking qemu / libvirt from accessing ISO

Question

I am unable to get selinux to allow libvirt to access images and ISO files outside of the default libvirt directory. What makes it more frustrating is that audit2allow and setroubleshooter does not see any issue even though I have a fail entry in audit.log

type=VIRT_CONTROL msg=audit(1576848063.439:6601): pid=1265 uid=0 auid=4294967295 ses=4294967295 \
subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=kvm op=start reason=booted vm="Unifi" uuid=37eed7bf-a37f-4d49-86c2-b9a6bb8682c3 \
vm-pid=-1 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=failed'UID="root" AUID="unset"

Image

-rw-------+ 1 root root system_u:object_r:svirt_image_t:s0            53695545344 Dec 20 08:31 unifi.qcow2

ISO

-rw-rwxr--+  1 qemu    qemu system_u:object_r:virt_content_t:s0                   851443712 Sep 29  2018  ubuntu-18.04.1-live-server-amd64.iso

My file_contexts.local file has the following entries that look fine to my untrained eye.

/data/libvirt(/.*)?    system_u:object_r:svirt_image_t:s0
/data/archive/ISO(/.*)?    system_u:object_r:svirt_image_t:s0

Thoughts?

EDIT 2 (update requested by A.B.):

SELinux output after enabling debugging.

type=AVC msg=audit(1577807557.017:10195): avc:  denied  { search } for  pid=13605 comm="qemu-kvm" name="/" dev="dm-8" ino=2 scontext=system_u:system_r:svirt_t:s0:c682,c798 tcontext=system_u:object_r:container_file_t:s0:c132,c155 tclass=dir permissive=0
    Was caused by:

#Constraint rule:

#   mlsconstrain dir { ioctl read lock search } ((h1 dom h2 -Fail-)  or (t1 != mcs_constrained_type -Fail-) ); Constraint DENIED

#   Possible cause is the source level (s0:c682,c798) and target level (s0:c132,c155) are different.

audit2allow -i /var/log/audit/audit.log -m qemu-kvm

module qemu-kvm 1.0;

require {
    type initrc_t;
    type container_file_t;
    type setroubleshootd_t;
    type NetworkManager_t;
    type svirt_t;
    type system_dbusd_t;
    class process { noatsecure rlimitinh siginh };
    class dir search;
    class capability net_admin;
}

#============= NetworkManager_t ==============
allow NetworkManager_t initrc_t:process { noatsecure rlimitinh siginh };

#============= svirt_t ==============

#!!!! This avc is a constraint violation.  You would need to modify the attributes of either the source or target types to allow this access.
#Constraint rule: 
#   mlsconstrain dir { ioctl read lock search } ((h1 dom h2 -Fail-)  or (t1 != mcs_constrained_type -Fail-) ); Constraint DENIED

#   Possible cause is the source level (s0:c682,c798) and target level (s0:c132,c155) are different.
allow svirt_t container_file_t:dir search;

#============= system_dbusd_t ==============
allow system_dbusd_t self:capability net_admin;
allow system_dbusd_t setroubleshootd_t:process { noatsecure rlimitinh siginh };

EDIT : Here is the error I am receiving. The label looks correct and the file permissions are set for qemu to rwx

Error starting domain: internal error: process exited while connecting to monitor: 2019-12-20T15:34:53.600905Z qemu-kvm: -drive file=/data/archive/ISO/ubuntu-18.04.1-live-server-amd64.iso,format=raw,if=none,id=drive-sata0-0-0,media=cdrom,readonly=on: Could not open '/data/archive/ISO/ubuntu-18.04.1-live-server-amd64.iso': Permission denied

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 111, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 66, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/domain.py", line 1279, in startup
    self._backend.create()
  File "/usr/lib/python3/dist-packages/libvirt.py", line 1080, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirt.libvirtError: internal error: process exited while connecting to monitor: 2019-12-20T15:34:53.600905Z qemu-kvm: -drive file=/data/archive/ISO/ubuntu-18.04.1-live-server-amd64.iso,format=raw,if=none,id=drive-sata0-0-0,media=cdrom,readonly=on: Could not open '/data/archive/ISO/ubuntu-18.04.1-live-server-amd64.iso': Permission denied

File permissions

getfacl /data/libvirt/images
getfacl: Removing leading '/' from absolute path names
# file: data/libvirt/images
# owner: qemu
# group: qemu
# flags: ss-
user::rwx
group::rwx
other::--x
default:user::rwx
default:user:qemu:rwx
default:group::rwx
default:group:qemu:rwx
default:mask::rwx
default:other::--x

Answer 1

Big thanks to dac.override and A.B for giving me what I needed to find this.

Checking the top level directory '/data' reveals a label of

system_u:object_r:container_file_t:s0:c132,c155

This also shows up on sub directories as well, including the '/data/libvirt' directory. This is caused by a container that runs daily used for backing up the '/data' directory. The container mounted this directory using

-v /data:/data:ro,Z

This is a habit I formed when working with production containers. Changing to

-v /data:/data:ro,z

allows the backup to run without libvirt being blocked. Now we get the following label

system_u:object_r:container_file_t:s0

Please note this is most likely not best practice. However, for my personal server it will do. All other containers and services will run from separate VMs with the host just being a host.

Answer 2

Standard SELinux troubleshoot procedure:

1. Can the issue be reproduced? If yes move to 2.
2. Does it work in permissive mode? If yes move to 3.
3. Does SELinux log event records? ausearch -m avc,user_avc,selinux_err -i. If yes interpret them and/or let audit2why interpret them for you. If no move to 4.
4. run semodule -DB to make SELinux verbose, then reproduce the issue and move back to 3.

If SELinux blocks then SELinux logs. You need to event records so that you can interpet the issue before you can implement a solution.

You have not provided any SELinux event logs, and so i can only speculate at this point. It may have to do with the labels of your custom /data mountpoint. If libvirt cannot traverse /data then it can never get to the images.

Answer 3

For those of you that do have relevant errors in audit.log, if this is a fresh install, you can use audit2allow to automatically generate the correct SELinux configurations for you. I personally only running this on fresh installs as you can potentially allow unauthorized software SELinux exceptions if you are not careful.

1. Make sure SELinux is running in Permissive mode.
2. Make sure denies are being logged in /var/log/audit.log. If nothing is present, run semodule -DB and run the offending program again; logs should be generated. Once they are, run semodule -B to disable verbose logging.
3. Run audit2allow -w -a - this will show you what is being blocked in a human-readable form.
4. Run audit2allow -a to view the Type Enforcement rule that allows the denied access
5. If you agree with the above output, run audit2allow -a -M newrules to create a custom module. The -M option creates a Type Enforcement file (.te) with the name specified with -M, in your current working directory
6. To install the module, run semodule -i newrules.pp
7. Set SELinux to Enforcing mode; test functionality.

I recommend only running this on new servers that are being built as you could potentially allow applications to have SELinux bypasses that are not required to have one. This can be dangerous in a production server. If you have multiple things that audit2allow mentions, you can use grep as mentioned in Red Hat's documentation. Please refer to the audit2allow link for the official documentation.