Driver for GTX 1080 doesn't work on guest when using KVM PCI Passthrough

Question

I believe we are running into a possible bug with the GTX 1080 (driver) and PCI Passthrough.

My host is an Ubuntu 14.04 system. My guest is an Ubuntu 14.04/16.04 system (both do the same thing).

I can see device inside the guest VM:

$ lspci -vnn | grep VGA
00:05.0 VGA compatible controller: NVIDIA Corporation Device 1b80 (rev a1)

I was able to successfully install the driver (370.2, latest driver). It installs, but is not recognized by nvidia-smi:

$ nvidia-smi 
Unable to determine the device handle for GPU 0000:00:05.0: Unknown Error

Looking in dmesg I see the following error message

[29.535583] nvidia 0000:00:05.0: irq 45 for MSI/MSI-X
[29.577727] NVRM: RmInitAdapter failed! (0x23:0x56:458)
[29.577807] NVRM: rm_init_adapter failed for device bearing minor number 0

I can switch out the GTX 1080 for a different card (M4000, do passthrough and install drivers on guest) and it works. I am going to try tomorrow with another Geforce card.

Another person on the NVidia forums had the SAME exact issue as me (but no answer).

Is there anyway to debug this further?

Answer 1

I had the same problem, I found the answer at https://www.evonide.com/non-root-gpu-passthrough-setup/. You need to add -cpu host,kvm=off to the qemu command line. I'm using ganeti, so the following fixed the problem:

gnt-instance modify -H cpu_type="host\,kvm=off"

If I understand correctly this flag does not switch off KVM acceleration for the guest, that's switched on with -machine pc,accel=kvm. But it switches off nested KVM acceleration for the guest (so you won't be able to run a KVM virtual machine inside the guest).

Answer 2

I don't know if I have much to contribute, it seems to me that nvidia is actively sabotaging pci passthrough attempts (you are supposed to buy a Quadro card).

Try faking the pci device ids by adding x-pci-vendor-id=0x10DE,x-pci-device-id=0x11BA,x-pci-sub-vendor-id=0x10DE,x-pci-sub-device-id=0x0965 to the -device vfio-pci, argument. This will make it pretend to be a Quadro K5000.

Answer 3

I fought this for a while, if the gpu is the only one in your system, you need to get grub and linux to leave it alone. I boot my machine with a serial console, BUT the key part is video=efifb:off

GRUB_CMDLINE_LINUX="console=tty1 console=ttyS0,115200" GRUB_TERMINAL="console serial" GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=1" GRUB_CMDLINE_LINUX="quiet splash video=efifb:off intel_iommu=on console=ttyS0,115200"

https://unix.stackexchange.com/questions/328534/driver-for-gtx-1080-doesnt-work-on-guest-when-using-kvm-pci-passthrough