Driver for GTX 1080 doesn't work on guest when using KVM PCI Passthrough

Question

I believe we are running into a possible bug with the GTX 1080 (driver) and PCI Passthrough.

My host is an Ubuntu 14.04 system. My guest is an Ubuntu 14.04/16.04 system (both do the same thing).

I can see device inside the guest VM:

lspci -vnn | grep VGA

00:05.0 VGA compatible controller: NVIDIA Corporation Device 1b80 (rev a1)

I was able to successfully install the driver (370.2, latest driver). It installs, but is not recognized by nvidia-smi:

nvidia-smi 

Unable to determine the device handle for GPU 0000:00:05.0: Unknown Error

Looking in dmesg I see the following error message

[ 29.535583] nvidia 0000:00:05.0: irq 45 for MSI/MSI-X [
29.577727] NVRM: RmInitAdapter failed! (0x23:0x56:458) [ 29.577807] NVRM: rm_init_adapter failed for device bearing minor number 0

I can switch out the GTX 1080 for a different card (M4000, do passthrough and install drivers on guest) and it works. I am going to try tomorrow with another Geforce card.

Another person on the NVidia forums had the SAME exact issue as me (but no answer).

https://devtalk.nvidia.com/default/topic/957757/gtx-1080-amp-kvm-pci-passthrough-to-guest/?offset=2#5033118

Is there anyway to debug this further?

Answer 1

I had the same problem, I found the answer at https://www.evonide.com/non-root-gpu-passthrough-setup/. You need to add -cpu host,kvm=off to the qemu command line. I'm using ganeti, so the following fixed the problem:

gnt-instance modify -H cpu_type="host\,kvm=off"

If I understand correctly this flag does not switch off KVM acceleration for the guest, that's switched on with -machine pc,accel=kvm. But it switches off nested KVM acceleration for the guest (so you won't be able to run a KVM virtual machine inside the guest).

Answer 2

I don't know if I have much to contribute, it seems to me that nvidia is actively sabotaging pci passthrough attempts (you are supposed to buy a Quadro card).

Try faking the pci device ids by adding x-pci-vendor-id=0x10DE,x-pci-device-id=0x11BA,x-pci-sub-vendor-id=0x10DE,x-pci-sub-device-id=0x0965 to the -device vfio-pci, argument. This will make it pretend to be a Quadro K5000.

Answer 3

I fought this for a while, if the gpu is the only one in your system, you need to get grub and linux to leave it alone. I boot my machine with a serial console, BUT the key part is video=efifb:off

GRUB_CMDLINE_LINUX="console=tty1 console=ttyS0,115200" GRUB_TERMINAL="console serial" GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1" GRUB_CMDLINE_LINUX="quiet splash video=efifb:off intel_iommu=on console=ttyS0,115200"

https://unix.stackexchange.com/questions/328534/driver-for-gtx-1080-doesnt-work-on-guest-when-using-kvm-pci-passthrough