2

I am unable to get selinux to allow libvirt to access images and ISO files outside of the default libvirt directory. What makes it more frustrating is that audit2allow and setroubleshooter does not see any issue even though I have a fail entry in audit.log

type=VIRT_CONTROL msg=audit(1576848063.439:6601): pid=1265 uid=0 auid=4294967295 ses=4294967295 \
subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=kvm op=start reason=booted vm="Unifi" uuid=37eed7bf-a37f-4d49-86c2-b9a6bb8682c3 \
vm-pid=-1 exe="/usr/sbin/libvirtd" hostname=? addr=? terminal=? res=failed'UID="root" AUID="unset"

Image

-rw-------+ 1 root root system_u:object_r:svirt_image_t:s0            53695545344 Dec 20 08:31 unifi.qcow2

ISO

-rw-rwxr--+  1 qemu    qemu system_u:object_r:virt_content_t:s0                   851443712 Sep 29  2018  ubuntu-18.04.1-live-server-amd64.iso

My file_contexts.local file has the following entries that look fine to my untrained eye.

/data/libvirt(/.*)?    system_u:object_r:svirt_image_t:s0
/data/archive/ISO(/.*)?    system_u:object_r:svirt_image_t:s0

Thoughts?

EDIT 2 (update requested by A.B.):

SELinux output after enabling debugging.

type=AVC msg=audit(1577807557.017:10195): avc:  denied  { search } for  pid=13605 comm="qemu-kvm" name="/" dev="dm-8" ino=2 scontext=system_u:system_r:svirt_t:s0:c682,c798 tcontext=system_u:object_r:container_file_t:s0:c132,c155 tclass=dir permissive=0
    Was caused by:

#Constraint rule:

#   mlsconstrain dir { ioctl read lock search } ((h1 dom h2 -Fail-)  or (t1 != mcs_constrained_type -Fail-) ); Constraint DENIED

#   Possible cause is the source level (s0:c682,c798) and target level (s0:c132,c155) are different.

audit2allow -i /var/log/audit/audit.log -m qemu-kvm

module qemu-kvm 1.0;

require {
    type initrc_t;
    type container_file_t;
    type setroubleshootd_t;
    type NetworkManager_t;
    type svirt_t;
    type system_dbusd_t;
    class process { noatsecure rlimitinh siginh };
    class dir search;
    class capability net_admin;
}

#============= NetworkManager_t ==============
allow NetworkManager_t initrc_t:process { noatsecure rlimitinh siginh };

#============= svirt_t ==============

#!!!! This avc is a constraint violation.  You would need to modify the attributes of either the source or target types to allow this access.
#Constraint rule: 
#   mlsconstrain dir { ioctl read lock search } ((h1 dom h2 -Fail-)  or (t1 != mcs_constrained_type -Fail-) ); Constraint DENIED

#   Possible cause is the source level (s0:c682,c798) and target level (s0:c132,c155) are different.
allow svirt_t container_file_t:dir search;

#============= system_dbusd_t ==============
allow system_dbusd_t self:capability net_admin;
allow system_dbusd_t setroubleshootd_t:process { noatsecure rlimitinh siginh };

EDIT : Here is the error I am receiving. The label looks correct and the file permissions are set for qemu to rwx

Error starting domain: internal error: process exited while connecting to monitor: 2019-12-20T15:34:53.600905Z qemu-kvm: -drive file=/data/archive/ISO/ubuntu-18.04.1-live-server-amd64.iso,format=raw,if=none,id=drive-sata0-0-0,media=cdrom,readonly=on: Could not open '/data/archive/ISO/ubuntu-18.04.1-live-server-amd64.iso': Permission denied

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 111, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 66, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/domain.py", line 1279, in startup
    self._backend.create()
  File "/usr/lib/python3/dist-packages/libvirt.py", line 1080, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirt.libvirtError: internal error: process exited while connecting to monitor: 2019-12-20T15:34:53.600905Z qemu-kvm: -drive file=/data/archive/ISO/ubuntu-18.04.1-live-server-amd64.iso,format=raw,if=none,id=drive-sata0-0-0,media=cdrom,readonly=on: Could not open '/data/archive/ISO/ubuntu-18.04.1-live-server-amd64.iso': Permission denied

File permissions

getfacl /data/libvirt/images
getfacl: Removing leading '/' from absolute path names
# file: data/libvirt/images
# owner: qemu
# group: qemu
# flags: ss-
user::rwx
group::rwx
other::--x
default:user::rwx
default:user:qemu:rwx
default:group::rwx
default:group:qemu:rwx
default:mask::rwx
default:other::--x
  • Try changing ownership of the directories that contain these files. I use /media/VM-testing/ISO-Images and /media/VM-testing/KVM, both owned by libvirt-qemu:kvm. I also changed the ownership on the ISOs as well, but the qcow2 disks are still owned by root. – ajgringo619 Dec 20 '19 at 16:51
  • @ajgringo619 I think I have done this. I do not have a libvirt-qemu user, just qemu. getfacl /data/libvirt/images getfacl: Removing leading '/' from absolute path names # file: data/libvirt/images # owner: qemu # group: qemu # flags: ss- user::rwx group::rwx other::--x default:user::rwx default:user:qemu:rwx default:group::rwx default:group:qemu:rwx default:mask::rwx default:other::--x – TurboAAA Dec 20 '19 at 17:55
  • Can you add your comment to your question? It's really hard to read. Also...what OS are you using? – ajgringo619 Dec 20 '19 at 17:57
  • Sure. PRETTY_NAME="CentOS Linux 8 (Core)" – TurboAAA Dec 20 '19 at 20:10
  • 1
    run semodule -DB to disable the "dontaudit" feature. This will probably create much more audit logs. See if you find something related to your issue. Enable back with semodule -B . That's as example in the man page. – A.B Dec 29 '19 at 2:04
1

Big thanks to dac.override and A.B for giving me what I needed to find this.

Checking the top level directory '/data' reveals a label of

system_u:object_r:container_file_t:s0:c132,c155

This also shows up on sub directories as well, including the '/data/libvirt' directory. This is caused by a container that runs daily used for backing up the '/data' directory. The container mounted this directory using

-v /data:/data:ro,Z

This is a habit I formed when working with production containers. Changing to

-v /data:/data:ro,z

allows the backup to run without libvirt being blocked. Now we get the following label

system_u:object_r:container_file_t:s0

Please note this is most likely not best practice. However, for my personal server it will do. All other containers and services will run from separate VMs with the host just being a host.

0
+50

Standard SELinux troubleshoot procedure:

  1. Can the issue be reproduced? If yes move to 2.
  2. Does it work in permissive mode? If yes move to 3.
  3. Does SELinux log event records? ausearch -m avc,user_avc,selinux_err -i. If yes interpret them and/or let audit2why interpret them for you. If no move to 4.
  4. run semodule -DB to make SELinux verbose, then reproduce the issue and move back to 3.

If SELinux blocks then SELinux logs. You need to event records so that you can interpet the issue before you can implement a solution.

You have not provided any SELinux event logs, and so i can only speculate at this point. It may have to do with the labels of your custom /data mountpoint. If libvirt cannot traverse /data then it can never get to the images.

  • I already added the selinux audit log output in my OP (very first code snippet). If you read through the other comments you would have seen that this is reproducible, does work in permissive mode, the logs were limiting/not helpful, and another user had already pointed out I need to enable additional logging. – TurboAAA Dec 31 '19 at 16:14
  • The context of the directory at inode 2 on device dm-8 does not allow this particular libvirt guest access. This is probably due to misconfiguration. Determine the exact location of the directory mentioned in the audit log: find / -inum 2. Then try to determine why it might be mislabeled. I think I have an idea. Are you sharing that location between some container solution and libvirt? The gist is that these technologies leverage SELinux to enforce separation based on categories so that containers and guests cannot interfere with eachother. Note how s0:c682,c798 and s0:c132,c155. – dac.override Jan 1 at 12:48
  • 1
    Separate your containers and your libvirt guests For example /data/myguests and /data/mycontainers Your libvirt guest is trying to access a directory that is in use by a container. The only way to allow that is to configure libvirt and your container manager to run the guest and container with the same categories. access.redhat.com/documentation/en-us/red_hat_enterprise_linux/… – dac.override Jan 1 at 13:00
  • Interesting observation, you might be onto something. Though this is a "new" server build, it is the third generation with the data directory being replicated from the previous. About four years ago I was playing with LXC, and it wouldn't be a stretch that I did something stupid with the labeling. I do run podman on this new host for backing up to backblaze, and on the old server I ran containers off a separate subdirectory under /data (/data used to be on a ZFS pool, now it's just a mirrored LVM pool) but this time I want to run the containers from a VM off the host. – TurboAAA Jan 1 at 15:23
  • dac.override, thank you very much for the insight. The directory /data was set to system_u:object_r:container_file_t:s0:c132,c155 as well as all subdirectories except /data/libvirt. This was caused by my backblaze backup container as I was mounting the entire '/data' directory into the container with -v /data:/data:ro,Z. – TurboAAA Jan 1 at 15:43
0

For those of you that do have relevant errors in audit.log, if this is a fresh install, you can use audit2allow to automatically generate the correct SELinux configurations for you. I personally only running this on fresh installs as you can potentially allow unauthorized software SELinux exceptions if you are not careful.

  1. Make sure SELinux is running in Permissive mode.
  2. Make sure denies are being logged in /var/log/audit.log. If nothing is present, run semodule -DB and run the offending program again; logs should be generated. Once they are, run semodule -B to disable verbose logging.
  3. Run audit2allow -w -a - this will show you what is being blocked in a human-readable form.
  4. Run audit2allow -a to view the Type Enforcement rule that allows the denied access
  5. If you agree with the above output, run audit2allow -a -M newrules to create a custom module. The -M option creates a Type Enforcement file (.te) with the name specified with -M, in your current working directory
  6. To install the module, run semodule -i newrules.pp
  7. Set SELinux to Enforcing mode; test functionality.

I recommend only running this on new servers that are being built as you could potentially allow applications to have SELinux bypasses that are not required to have one. This can be dangerous in a production server. If you have multiple things that audit2allow mentions, you can use grep as mentioned in Red Hat's documentation. Please refer to the audit2allow link for the official documentation.

  • I am no selinux expert, but I fully endorse this recommendation and the accompanying warnings. Sadly it does not help my issue. – TurboAAA Jan 1 at 15:25
  • @TurboAAA I didn't figure it would per your comments, but it would have been a good help to me a few times in the past, so I figured this was a good place to add it – cutrightjm Jan 2 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.