GPU passthrough works with UEFI firmware but not Windows iso

Question

I am trying to set up a virtual machine with passthrough graphics. I am able to get the passthrough working for the UEFI shell, but not the official Windows installer. I can get the Windows installer to work, but only with emulated graphics

This boots the windows installer in a QEMU Window:

sudo qemu-system-x86_64 --enable-kvm \
-name TESTVM,process=TESTVM \
-cdrom /media/big-tank-8TB/OSISOS/Windows/WindowsOct2018.iso \
-smp 4 \
-cpu core2duo \
-m 4096 \
-vga qxl

This also boots the windows installer in a QEMU window (still no passthrough)

sudo qemu-system-x86_64 --enable-kvm \
-name TESTVM,process=TESTVM \
-cdrom /media/big-tank-8TB/OSISOS/Windows/WindowsOct2018.iso \
-smp 4 \
-cpu core2duo \
-m 4096 \
-device vfio-pci,host=43:00.0,multifunction=on \
-device vfio-pci,host=43:00.1 

But if I specify the paths to UEFI firmware, I get the Tiano slpash screen and then the UEFI shell both on the monitor attached to my passed-through video card and in a QEMU window.

sudo qemu-system-x86_64 --enable-kvm \
-name TESTVM,process=TESTVM \
-cdrom /media/big-tank-8TB/OSISOS/Windows/WindowsOct2018.iso \
-smp 4 \
-cpu core2duo \
-m 4096 \
-device vfio-pci,host=43:00.0,multifunction=on \
-device vfio-pci,host=43:00.1 \
-drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \
-drive if=pflash,format=raw,file=/usr/share/OVMF/OVMF_VARS.fd 

Why is the passthrough working only in the absence of the UEFI files? Or, why is specifying the UEFI files preventing me from starting Windows?

Edit: Tried downloading a different version of Windows (April 2018 instead of the October one), same problem.

Edit: Tried purging and reinstalling OVMF, but no luck.

Edit: I can get to the boot manager by typing "exit" in the shell, but selecting the available DVD drive (and all other options) immediately falls back to the boot manager.

Edit: Ran this:

-name TESTVM,process=TESTVM \
-drive file=/media/big-tank-8TB/OSISOS/Windows/Win10_1803_English_x64.iso,index=1,media=cdrom  \
-drive file=/media/big-tank-8TB/OSISOS/Windows/virtio-win-0.1.160.iso,index=2,media=cdrom \
-smp 4 \
-cpu core2duo \
-m 4096 \
-device vfio-pci,host=43:00.0,multifunction=on \
-device vfio-pci,host=43:00.1 \
-drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \
-drive if=pflash,format=raw,file=/usr/share/OVMF/OVMF_VARS.fd 

When I got too the uefi shell, I typed "exit" to get to the boot manager. In the boot manager, selecting the available DVD drive instantly fell back to the boot manager.

I then added another DVD drive by Boot Maintenance Manager > Boot Options > Add boot option, and then selected that in the Boot Manager menu. .

This gave me a very brief "press any key to boot from CD". If I am fast enough, this boots into the Windows installer BUT ONLY IN THE QEMU WINDOW. The screen attached to the passed-through card was black with a simple cursor, as opposed to mirroring as with the UEFI/Boot Manager.

Edit: I am trying to pass through an NVIDIA GTX1070. Mobo is ASRock x399 Taichi, CPU is Threadripper 1950X. OS is Ubuntu Server with XFCE installed.

Edit: If I proceed with the installation, I still have Windows in the QEMU window and just a TianoCore splash screen on the passthrough card. If I go to the device manager, Windows sees the card, but it is stopped for some reason.

Edit: I tried using these instructions to get rid of the code 43, to no avail. In order to try this, I used virt-install instead of qemu-system, and when doing this there is no TianoCore splash screen. But still code 43 when I get into Windows.

Edit: used dmesg to check for memory reservation errors as described here. Found none. Edit: Also from the above link, used ROM parser and confirmed the presence of a "type 3 (EFI)"

Answer 1

You are on the correct track already. GPU Passthrough is not perfect, especially if it's an NVidia Card (Which you don't mention NVidia or AMD). Finish the setup on the Qemu Window. Make sure the Windows Machine is connected to Internet and let Windows Update install the graphics drivers for you. When you come back you should be greeted by a second monitor, if not, reboot. I usually then remove the spice/vnc console and only have the GPU monitor attached. Getting GPU Passthrough to work is all about trial and error.

Other things to try:

• Install Windows without GPU passthrough, then attempt to passthrough GPU.
• Install drivers via NVidia_drivers.exe
• Install drivers via Windows Update
• Bios vs UEFI
• Q35 vs i440fx

Note: Code 43 is a known error w/ NVidia relating to NVidia drivers checking if they are running in a VM. NVidia sells cards specifically for running in a VM environment and attempts to block installation of drivers for consumer grade cards in a VM. You need to make sure to use the following in your domain.xml

<kvm>
 <hidden state='on'/>
</kvm>

See https://passthroughpo.st/apply-error-43-workaround/ and other resources for examples.

Here is a screenshot of my config:

Here is the "relevant" part of my domain.xml, I can share entire thing if you want, but it's got a bunch of unnecessary things.

  <os>
    <type arch='x86_64' machine='pc-i440fx-2.10'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/Windows10_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
  </features>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>

Answer 2

I did it with this:

sudo qemu-system-x86_64 --enable-kvm \
-name TESTVM,process=TESTVM \
-drive id=disk0,if=virtio,cache=none,format=raw,file=/dev/nvme-tank/ntfs-zvol,index=2 \
-smp 4 \
-cpu core2duo,kvm=off \
-m 4096 \
-device vfio-pci,host=43:00.0,multifunction=on \
-device vfio-pci,host=43:00.1 \
-drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \
-drive if=pflash,format=raw,file=/usr/share/OVMF/OVMF_VARS.fd 

and then going to the GTX 1070 entry in Device Manager and updating the drivers. Last time (using virt-install) I installed updates from the NVIDIA website. Now to get this working with virt-install so that I can have an editable domain XML.

Edit: So far, when using virt-install, I now (again) get the Tiano screen, but this time installing the drivers as described above was not sufficient, nor did it work to import the above install using virt-install (on a side note, it took me a while to realize that all I had to do to make virt-install work with my existing installation was to include the boot iso, allowing it to boot from the installation without error).

Edit: So, I think I have it working now.

1. Do the initial install using the above qemu-system-x86_64
2. Updated drivers using Device Manager

3. Then do virt-install (but get error 43 again):

   virt-install \ --name myWINVM1 \ --boot uefi \ --ram 4096 \ --features kvm_hidden=on \ --hostdev 43:00.0,address.type=pci,address.multifunction=on \ --hostdev 43:00.1,address.type=pci \ --disk path=/dev/nvme-tank/ntfs-zvol-part2 \ --cpu core2duo \ --vcpus 4 \ --os-type windows \ --os-variant win10 \ --network bridge=virbr0 \ --console pty,target_type=serial \ --disk /media/big-tank-8TB/OSISOS/Windows/WindowsOct2018.iso,device=cdrom \ --disk /media/big-tank-8TB/OSISOS/Windows/virtio-win-0.1.160.iso,device=cdrom

4. Deleted the spice graphics device from the myWINVM1 XML. Still couldnt get past TianoCore.

5. added the fake vendor ID line to the my XML

6. Added my USB devices

my final xml:

<domain type='kvm' id='24'>
  <name>myWINVM1</name>
  <uuid>43e052b3-b3da-4025-92d8-ec7c8ff96ae9</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-bionic'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/myWINVM1_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='1234567890ab'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>core2duo</model>
    <feature policy='disable' name='ss'/>
    <feature policy='disable' name='monitor'/>
    <feature policy='require' name='cx16'/>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='lahf_lm'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/dev/nvme-tank/ntfs-zvol-part2'/>
      <backingStore/>
      <target dev='hda' bus='ide'/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/media/big-tank-8TB/OSISOS/Windows/WindowsOct2018.iso'/>
      <backingStore/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <alias name='ide0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/media/big-tank-8TB/OSISOS/Windows/virtio-win-0.1.160.iso'/>
      <backingStore/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:36:1d:c4'/>
      <source bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='rtl8139'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/4'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/4'>
      <source path='/dev/pts/4'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <sound model='ich6'>
      <alias name='sound0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc33b'/>
        <address bus='5' device='3'/>
      </source>
      <alias name='hostdev0'/>
      <address type='usb' bus='0' port='4'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc52b'/>
        <address bus='5' device='4'/>
      </source>
      <alias name='hostdev1'/>
      <address type='usb' bus='0' port='5'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x43' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x43' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir0'/>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir1'/>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='apparmor' relabel='yes'>
    <label>libvirt-43e052b3-b3da-4025-92d8-ec7c8ff96ae9</label>
    <imagelabel>libvirt-43e052b3-b3da-4025-92d8-ec7c8ff96ae9</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+64055:+120</label>
    <imagelabel>+64055:+120</imagelabel>
  </seclabel>
</domain>

Answer 3

This is what worked for me after a long struggle:

• first of all edit the qemu VM configuration file this way:

   cd /etc/libvirt/qemu    
   sudo virsh define win10_GPU.xml
   sudo virsh edit win10_GPU
  

  where "win10_GPU" is your virtual machine name of course and add
  

   <qemu:commandline>
     <qemu:arg value='-cpu'/>
     <qemu:arg value='host,hv_time,kvm=off,hv_vendor_id=null,-hypervisor'/>   
   </qemu:commandline>
  

  after </devices> almost at the end of the file
  then hit ctrl+O (save), ctrl+X (exit) and type sudo virsh define win10_GPU.xml again;
  
  

• remove every other display interface from your virtual machine before to even start install windows (like spice, video cirrus, whatever);
  
  

• Do not install the 18xx build of windows10. I installed a 16xx version which is working perfectly even after the full updates.

This is my personal experience. Any other thing I tried led me to that error code 43.