Passthrough Sonnet USB-C card to Windows guest via KVM and vfio

Question

I am trying to passthrough a USB card to a guest, but I can't seem to make the VFIO driver stick, instead it seems that the host loads and takes over the card. I have a GPU successfully passed through, so I am sure that vfio/IOMMU are working.

card: Sonnet Allegro USB 3.1, Two-Port USB-C 10Gb PCIe Card (USB3C-2PM-E) motherboard: ASRock x399 Taichi CPU: Threadripper 1950x I tried following the same procedure that I used to passthrough a GPU, which I mostly learned here:

First, get the ID of the card:

~ lspci | grep USB
01:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43ba (rev 02)
0a:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller
0b:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller
45:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller

Next. get the vendor, product ID, and an alias, then modify system files accordingly in hopes that it will prevent the card from being grabbed by the host:

~ lspci -nn | grep 0a:00
0a:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
cat /sys/bus/pci/devices/0000:0a:00.0/modalias
pci:v00001B21d00001242sv000016B8sd00007230bc0Csc03i30
~ sudo nano /etc/modprobe.d/local.conf

paste at end:

alias pci:v00001B21d00001242sv000016B8sd00007230bc0Csc03i30 vfio-pci
options vfio-pci ids=1b21:1242

then:

~ sudo update-initramfs -u

and reboot and check if the card is using the vfio driver:

~ lspci -kn | grep -A 2 0a
0a:00.0 0c03: 1b21:1242
    Subsystem: 16b8:7230
    Kernel driver in use: xhci_hcd

I also tried making an override script as per the same blog mentioned above:

sudo nano /sbin/vfio-pci-override-usb.sh


#!/bin/sh

DEVS="0000:0a:00.0"

for DEV in $DEVS; do
echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done

modprobe -i vfio-pci

#####

sudo chmod 755 /sbin/vfio-pci-override-usb.sh

sudo nano /etc/modprobe.d/local.conf

commented out previous entry and added:

install vfio-pci /sbin/vfio-pci-override-usb.sh
sudo update-initramfs -u
reboot

but still no good:

 ~ lspci -kn | grep -A 2 0a:00
0a:00.0 0c03: 1b21:1242
    Subsystem: 16b8:7230
    Kernel driver in use: xhci_hcd

also: When I look for the driver, in lspci (to see what else is using it), it brings up a bunch of devices but not 0a:00.

~ lspci -kn | grep -A 2 xhci
    Kernel driver in use: xhci_hcd
01:00.1 0106: 1022:43b6 (rev 02)
    Subsystem: 1b21:1062
--
    Kernel driver in use: xhci_hcd
0b:00.0 1300: 1022:145a
    Subsystem: 1022:145a
--
    Kernel driver in use: xhci_hcd
0c:00.0 1300: 1022:1455
    Subsystem: 1022:1455
--
    Kernel driver in use: xhci_hcd
46:00.0 1300: 1022:1455
    Subsystem: 1022:1455

Next I tried Tried blacklisting the driver:

sudo nano /etc/default/grub

# original: GRUB_CMDLINE_LINUX_DEFAULT=”amd_iommu=on”
# to: 
GRUB_CMDLINE_LINUX_DEFAULT=”modprobe.blacklist=xhci_hcd amd_iommu=on”

##then:
 sudo update-grub

 ##Uncomment in local.conf:
# options vfio-pci ids=1b21:1242

 sudo update-initramfs -u

still no:

lspci -kn | grep -A 2 0a:00
0a:00.0 0c03: 1b21:1242
    Subsystem: 16b8:7230
    Kernel driver in use: xhci_hcd

What else might I try? Am I correct that the kernel driver in use for that device must be vfio-pci?

Answer 1

I know I am a bit late to the party but I hope the following can help people on google quests facing similar issues. Please note I have only tested this on discrete USB controllers.

xhci_hcd is indeed built in the kernel so it claims the USB controller and unless a workaround is found vfio-pci is not assigned as the kernel in use. One way to do that is to invoke a script to bind/unbind

Another simpler way to solve this issue is to use the following command:

sudo driverctl -v set-override 0000:0X:00 vfio-pci

Where 0000:0X:00 is your pci id.

Upon reboot the driver assigned will be automatically be vfio-pci.

Answer 2

Haha silly me, it seems I went through a lot of trouble for nothing, I could just add the card in virt-install and it appears to be fine even though the card isn't bound to the VFIO driver.

EDIT: It seems I spoke too soon because I can't passthrough my NVIDIA GPU and and the USB card at the same time (virt-manager just hangs without error when I try to start it).. but I guess that is a separate question since it's about two specific cards together.

EDIT #2: The trick was to install Windows with a different GPU and then setup the GPU passthrough by editing the domain XML.