0

After successfully using an NVIDIA GTX 1070 passed though to a Windows guest under KVM, I now want to pass it to a linux guest, and that is proving to be a problem.

First, I tried creating a VM with the passthrough defined (43:00 is the NVIDIA card), but that results in the host completely freezing:

sudo virt-install \
--name ubuntu1 \
--ram 8192 \
--boot uefi \
--graphics vnc \
--machine q35 \
--disk path=/home/boss/VM_IMGs/ubuntu1.img,bus=virtio,size=40 \
--vcpus 4 \
--os-type linux \
--os-variant generic \
--network bridge=br0 \
--hostdev 0b:00.3,address.type=pci \
--hostdev 43:00.0,address.type=pci,address.multifunction=on \
--hostdev 43:00.1,address.type=pci \
--console pty,target_type=serial \
--cdrom /tmp/ubuntu-18.04.2-desktop-amd64.iso 

If I create and install the VM without the PCI passthrough and then add to the domain XML, the host also freezes. I used the instructions here https://orenbell.com/?p=228 when I added the following to my domain XML file:

<features>
  <acpi/>
  <apic/>
  <hyperv>
    <vendor_id state='on' value='1234567890ab'/>
  </hyperv>
  <kvm>
    <hidden state='on'/>
  </kvm>
  <vmport state='off'/>
</features>

and

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x43' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev2'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x43' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev3'/>
    </hostdev>

The freeze happens instantaneously when I click start in virt-manager. This is journalctl near the time of the crash:

I'm a bit worried because the current x370 Taichi BIOS breaks NVIDIA passthrough. My motherboard is an x399 Taichi, so I am worried that it might have the same problem. Below is journalctl from right before the crash when I start the VM with the card added.

Jul 15 05:22:02 virtland kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000048
Jul 15 05:22:02 virtland kernel: #PF error: [WRITE]
Jul 15 05:22:02 virtland kernel: PGD 0 P4D 0 
Jul 15 05:22:02 virtland kernel: Oops: 0002 [#1] PREEMPT SMP NOPTI
Jul 15 05:22:02 virtland kernel: CPU: 7 PID: 2796 Comm: libvirtd Tainted: P        W  OE     5.1.16-arch1-1-ARCH #1
Jul 15 05:22:02 virtland kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X399 Taichi, BIOS P3.50 12/24/2018
Jul 15 05:22:02 virtland kernel: RIP: 0010:mutex_lock+0x10/0x20
Jul 15 05:22:02 virtland kernel: Code: ab e8 44 b3 7e ff 0f 1f 40 00 0f 1f 44 00 00 be 02 00 00 00 e9 e1 fa ff ff 90 0f 1f 44 00 00 31 c0 65 48 8b 14 25 00 5c 01 00 <f0> 48 0f b1 17 75 01 c3 eb d6 66 0f 1f 44 00 00 0f 1f 44 00 00 41
Jul 15 05:22:02 virtland kernel: RSP: 0018:ffffab0750813aa0 EFLAGS: 00010246
Jul 15 05:22:02 virtland kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Jul 15 05:22:02 virtland kernel: RDX: ffff98b697440000 RSI: ffffab0750813985 RDI: 0000000000000048
Jul 15 05:22:02 virtland kernel: RBP: ffff98b8f559b000 R08: 0000000000000000 R09: ffffffffc34874d0
Jul 15 05:22:02 virtland kernel: R10: ffff98b6a9ece420 R11: ffff98b69bffb3e0 R12: 0000000000000048
Jul 15 05:22:02 virtland kernel: R13: ffffab0750813c40 R14: ffff98b8f5a50600 R15: ffff98b6d62de200
Jul 15 05:22:02 virtland kernel: FS:  00007f80597d3700(0000) GS:ffff98b8fd1c0000(0000) knlGS:0000000000000000
Jul 15 05:22:02 virtland kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 15 05:22:02 virtland kernel: CR2: 0000000000000048 CR3: 0000001d97d42000 CR4: 00000000003406e0
Jul 15 05:22:02 virtland kernel: Call Trace:
Jul 15 05:22:02 virtland kernel:  nouveau_bo_move_m2mf.constprop.0+0x14d/0x1f0 [nouveau]
Jul 15 05:22:02 virtland kernel:  nouveau_bo_move+0xa4/0x420 [nouveau]
Jul 15 05:22:02 virtland kernel:  ? nvif_vmm_unmap+0x38/0x60 [nouveau]
Jul 15 05:22:02 virtland kernel:  ? nouveau_vma_unmap+0x20/0x30 [nouveau]
Jul 15 05:22:02 virtland kernel:  ttm_bo_handle_move_mem+0x10b/0x4f0 [ttm]
Jul 15 05:22:02 virtland kernel:  ttm_bo_evict+0x150/0x1d0 [ttm]
Jul 15 05:22:02 virtland kernel:  ? nvkm_mm_free+0xbe/0x180 [nouveau]
Jul 15 05:22:02 virtland kernel:  ? drm_vma_offset_add+0x53/0x60 [drm]
Jul 15 05:22:02 virtland kernel:  ? ldsem_down_read+0x192/0x200
Jul 15 05:22:02 virtland kernel:  ttm_mem_evict_first+0x194/0x210 [ttm]
Jul 15 05:22:02 virtland kernel:  ttm_bo_force_list_clean+0x9f/0x170 [ttm]
Jul 15 05:22:02 virtland kernel:  ttm_bo_clean_mm+0x89/0xb0 [ttm]
Jul 15 05:22:02 virtland kernel:  nouveau_ttm_fini+0x2b/0xa0 [nouveau]
Jul 15 05:22:02 virtland kernel:  nouveau_drm_device_fini+0x72/0xd0 [nouveau]
Jul 15 05:22:02 virtland kernel:  nouveau_drm_device_remove+0x56/0x90 [nouveau]
Jul 15 05:22:02 virtland kernel:  pci_device_remove+0x3b/0xa0
Jul 15 05:22:02 virtland kernel:  device_release_driver_internal+0xe4/0x1d0
Jul 15 05:22:02 virtland kernel:  unbind_store+0xf5/0x120
Jul 15 05:22:02 virtland kernel:  kernfs_fop_write+0x10e/0x190
Jul 15 05:22:02 virtland kernel:  vfs_write+0xac/0x1b0
Jul 15 05:22:02 virtland kernel:  ksys_write+0x57/0xd0
Jul 15 05:22:02 virtland kernel:  do_syscall_64+0x5b/0x190
Jul 15 05:22:02 virtland kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 15 05:22:02 virtland kernel: RIP: 0033:0x7f805e0af9cf
Jul 15 05:22:02 virtland kernel: Code: 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 f9 fc ff ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2d 44 89 c7 48 89 44 24 08 e8 2c fd ff ff 48
Jul 15 05:22:02 virtland kernel: RSP: 002b:00007f80597d2400 EFLAGS: 00000293 ORIG_RAX: 0000000000000001
Jul 15 05:22:02 virtland kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f805e0af9cf
Jul 15 05:22:02 virtland kernel: RDX: 000000000000000c RSI: 00007f804c06e950 RDI: 0000000000000015
Jul 15 05:22:02 virtland kernel: RBP: 000000000000000c R08: 0000000000000000 R09: 00007f804d653620
Jul 15 05:22:02 virtland kernel: R10: 0000000000000000 R11: 0000000000000293 R12: 00007f804c06e950
Jul 15 05:22:02 virtland kernel: R13: 0000000000000000 R14: 0000000000000015 R15: 00007f80597d24a0
Jul 15 05:22:02 virtland kernel: Modules linked in: ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter 8021q garp mrp bridge stp llc nct6775 hwmon_vid amd64_edac_mod edac_mce_amd kvm_amd nls_iso8859_1 nls_cp437 vfat fat ext4 crc32c_generic mbcache jbd2 wmi_bmof arc4 nouveau kvm iwlmvm snd_hda_codec_realtek mac80211 snd_hda_codec_generic mxm_wmi ledtrig_audio crct10dif_pclmul snd_hda_codec_hdmi ttm ghash_clmulni_intel btusb snd_hda_intel zfs(POE) btrtl drm_kms_helper btbcm iwlwifi mousedev btintel joydev snd_hda_codec drm bluetooth aesni_intel snd_hda_core input_leds aes_x86_64 crypto_simd snd_hwdep cfg80211 cryptd agpgart glue_helper snd_pcm ecdh_generic vfio_pci igb crc16 snd_timer syscopyarea zunicode(POE) sysfillrect snd vfio_virqfd sp5100_tco zavl(POE) vfio_iommu_type1 ccp soundcore vfio sysimgblt i2c_algo_bit k10temp pcspkr fb_sys_fops dca i2c_piix4 icp(POE) rng_core rfkill irqbypass zlua(POE) wmi pcc_cpufreq evdev pinctrl_amd gpio_amdpt mac_hid acpi_cpufreq zcommon(POE) znvpair(POE)
Jul 15 05:22:02 virtland kernel:  spl(OE) ip_tables x_tables f2fs dm_snapshot dm_bufio dm_mod sr_mod cdrom sd_mod hid_generic usbhid hid uas usb_storage crc32_pclmul crc32c_intel ahci libahci libata xhci_pci xhci_hcd scsi_mod
Jul 15 05:22:02 virtland kernel: CR2: 0000000000000048
Jul 15 05:22:02 virtland kernel: ---[ end trace 7f898594892d1210 ]---
Jul 15 05:22:02 virtland kernel: RIP: 0010:mutex_lock+0x10/0x20
Jul 15 05:22:02 virtland kernel: Code: ab e8 44 b3 7e ff 0f 1f 40 00 0f 1f 44 00 00 be 02 00 00 00 e9 e1 fa ff ff 90 0f 1f 44 00 00 31 c0 65 48 8b 14 25 00 5c 01 00 <f0> 48 0f b1 17 75 01 c3 eb d6 66 0f 1f 44 00 00 0f 1f 44 00 00 41
Jul 15 05:22:02 virtland kernel: RSP: 0018:ffffab0750813aa0 EFLAGS: 00010246
Jul 15 05:22:02 virtland kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
Jul 15 05:22:02 virtland kernel: RDX: ffff98b697440000 RSI: ffffab0750813985 RDI: 0000000000000048
Jul 15 05:22:02 virtland kernel: RBP: ffff98b8f559b000 R08: 0000000000000000 R09: ffffffffc34874d0
Jul 15 05:22:02 virtland kernel: R10: ffff98b6a9ece420 R11: ffff98b69bffb3e0 R12: 0000000000000048
Jul 15 05:22:02 virtland kernel: R13: ffffab0750813c40 R14: ffff98b8f5a50600 R15: ffff98b6d62de200
Jul 15 05:22:02 virtland kernel: FS:  00007f80597d3700(0000) GS:ffff98b8fd1c0000(0000) knlGS:0000000000000000
Jul 15 05:22:02 virtland kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 15 05:22:02 virtland kernel: CR2: 0000000000000048 CR3: 0000001d97d42000 CR4: 00000000003406e0

One thing I noticed (which still should not cause a host crash, but does explain the passthrough not working) is that the device was still bound to nouveau drivers instead of vfio drivers. This is in spite of adding the appropriate alias entries to /etc/modprobe.d/vfio.conf :

alias pci:v000010DEd00001B81sv00003842sd00005173bc03sc00i00 vfio-pci
alias pci:v000010DEd000010F0sv00003842sd00005173bc04sc03i00 vfio-pci
options vfio-pci ids=10de:1b81,10de:10f0
#options vfio-pci disable_vga=1

I also looked at this tutorial: but that went wrong at this step, as the device was under nouveau, not xhci_hcd. Attempting to unbind the card from Nouveau caused the system to freeze:

echo '0000:43:00.0' > /sys/bus/pci/drivers/nouvaeu/unbind

The output of jctl was very similar to what I included above, from the other crash.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.