PCI Passthrough KVM with vfio-pci giving EINVAL on PCI Probe

Question

I am attempting to attach PCIe devices in passthrough to my KVM domain. I have two devices in one IOMMU group (#28):

03:00.0 PCI bridge: Texas Instruments XIO2001 PCI Express-to-PCI Bridge
04:00.0 Multimedia audio controller: Texas Instruments TMS320C6414 TMS320C6415 TMS320C6416

and lspci -n values:

03:00.0 0604: 104c:8240
04:00.0 0401: 104c:a106

I am running Ubuntu 14.04. I have:

• Installed KVM, set up domains, ensured KVM and vfio-pci modules are loaded
• Set intel_iommu=on on the kernel command line and verified it is enabled with dmesg
• kvm-ok prints acceleration is all set, VT-d is supported on the chipset
• Verified that the 03:00.0 device does not have a driver currently listed under ls -l /sys/bus/pci/devices/0000\:03\:00.0/

So I do this to bind the devices to vfio-pci:

# echo "104c a106" > /sys/bus/pci/drivers/vfio-pci/new_id
# echo "104c 8240" > /sys/bus/pci/drivers/vfio-pci/new_id

Then:

root@tupac:~# echo "0000:03:00.0" > /sys/bus/pci/drivers/vfio-pci/bind 
bash: echo: write error: No such device

However, the multimedia controller appears AOK:

root@tupac:~# ls -l /sys/bus/pci/drivers/vfio-pci/
total 0
lrwxrwxrwx 1 root root    0 Feb 24 13:07 0000:04:00.0 -> ../../../../devices/pci0000:00/0000:00:02.2/0000:03:00.0/0000:04:00.0
--w------- 1 root root 4096 Feb 24 13:05 bind
lrwxrwxrwx 1 root root    0 Feb 24 13:07 module -> ../../../../module/vfio_pci
--w------- 1 root root 4096 Feb 24 13:15 new_id
--w------- 1 root root 4096 Feb 23 19:47 remove_id
--w------- 1 root root 4096 Feb 23 19:43 uevent
--w------- 1 root root 4096 Feb 24 13:07 unbind

03:00.0 (which is a PCIe to PCI adapter) is not showing up under vfio-pci's devices. Likewise when attempting to use the kvm managed configuration I get:

virsh # start rivendell
error: Failed to start domain rivendell
error: Failed to bind PCI device '0000:03:00.0' to vfio-pci: No such device

and under syslog I get what I believe to be the PCIe EINVAL code:

Feb 23 19:46:46 tupac kernel: [  214.159625] vfio-pci: probe of 0000:03:00.0 failed with error -22

What is causing this? Is this some issue with the presence of the PCI bus in the system behind the adapter?

Answer 1

vfio-pci does not support bridges.

Contrary to what is suggested about the iommu group needing to be assigned together, the appropriate thing to do is to assign solely the device on the other side of the bridge, and then it shows up just fine in the guest.

Answer 2

This forum led me to this github repo which contains the script vfio-pci-bind.sh which fixed this issue for me when trying to bind a second wireless card to the vfio-pci driver.

The syntax is vfio-pci-bind.sh 0000:00:00.0 where 0000:00:00.0 is your device ID.