5

I am trying to set up a virtual machine with passthrough graphics. I am able to get the passthrough working for the UEFI shell, but not the official Windows installer. I can get the Windows installer to work, but only with emulated graphics

This boots the windows installer in a QEMU Window:

sudo qemu-system-x86_64 --enable-kvm \
-name TESTVM,process=TESTVM \
-cdrom /media/big-tank-8TB/OSISOS/Windows/WindowsOct2018.iso \
-smp 4 \
-cpu core2duo \
-m 4096 \
-vga qxl

This also boots the windows installer in a QEMU window (still no passthrough)

sudo qemu-system-x86_64 --enable-kvm \
-name TESTVM,process=TESTVM \
-cdrom /media/big-tank-8TB/OSISOS/Windows/WindowsOct2018.iso \
-smp 4 \
-cpu core2duo \
-m 4096 \
-device vfio-pci,host=43:00.0,multifunction=on \
-device vfio-pci,host=43:00.1 

But if I specify the paths to UEFI firmware, I get the Tiano slpash screen and then the UEFI shell both on the monitor attached to my passed-through video card and in a QEMU window.

sudo qemu-system-x86_64 --enable-kvm \
-name TESTVM,process=TESTVM \
-cdrom /media/big-tank-8TB/OSISOS/Windows/WindowsOct2018.iso \
-smp 4 \
-cpu core2duo \
-m 4096 \
-device vfio-pci,host=43:00.0,multifunction=on \
-device vfio-pci,host=43:00.1 \
-drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \
-drive if=pflash,format=raw,file=/usr/share/OVMF/OVMF_VARS.fd 

Why is the passthrough working only in the absence of the UEFI files? Or, why is specifying the UEFI files preventing me from starting Windows?

Edit: Tried downloading a different version of Windows (April 2018 instead of the October one), same problem.

Edit: Tried purging and reinstalling OVMF, but no luck.

Edit: I can get to the boot manager by typing "exit" in the shell, but selecting the available DVD drive (and all other options) immediately falls back to the boot manager.

Edit: Ran this:

-name TESTVM,process=TESTVM \
-drive file=/media/big-tank-8TB/OSISOS/Windows/Win10_1803_English_x64.iso,index=1,media=cdrom  \
-drive file=/media/big-tank-8TB/OSISOS/Windows/virtio-win-0.1.160.iso,index=2,media=cdrom \
-smp 4 \
-cpu core2duo \
-m 4096 \
-device vfio-pci,host=43:00.0,multifunction=on \
-device vfio-pci,host=43:00.1 \
-drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \
-drive if=pflash,format=raw,file=/usr/share/OVMF/OVMF_VARS.fd 

When I got too the uefi shell, I typed "exit" to get to the boot manager. In the boot manager, selecting the available DVD drive instantly fell back to the boot manager.

I then added another DVD drive by Boot Maintenance Manager > Boot Options > Add boot option, and then selected that in the Boot Manager menu. the boot entry I picked.

This gave me a very brief "press any key to boot from CD". If I am fast enough, this boots into the Windows installer BUT ONLY IN THE QEMU WINDOW. The screen attached to the passed-through card was black with a simple cursor, as opposed to mirroring as with the UEFI/Boot Manager.

Edit: I am trying to pass through an NVIDIA GTX1070. Mobo is ASRock x399 Taichi, CPU is Threadripper 1950X. OS is Ubuntu Server with XFCE installed.

Edit: If I proceed with the installation, I still have Windows in the QEMU window and just a TianoCore splash screen on the passthrough card. If I go to the device manager, Windows sees the card, but it is stopped for some reason. Error 43

Edit: I tried using these instructions to get rid of the code 43, to no avail. In order to try this, I used virt-install instead of qemu-system, and when doing this there is no TianoCore splash screen. But still code 43 when I get into Windows.

Edit: used dmesg to check for memory reservation errors as described here. Found none. Edit: Also from the above link, used ROM parser and confirmed the presence of a "type 3 (EFI)"

5
+50

You are on the correct track already. GPU Passthrough is not perfect, especially if it's an NVidia Card (Which you don't mention NVidia or AMD). Finish the setup on the Qemu Window. Make sure the Windows Machine is connected to Internet and let Windows Update install the graphics drivers for you. When you come back you should be greeted by a second monitor, if not, reboot. I usually then remove the spice/vnc console and only have the GPU monitor attached. Getting GPU Passthrough to work is all about trial and error.

Other things to try:

  • Install Windows without GPU passthrough, then attempt to passthrough GPU.
  • Install drivers via NVidia_drivers.exe
  • Install drivers via Windows Update
  • Bios vs UEFI
  • Q35 vs i440fx

Note: Code 43 is a known error w/ NVidia relating to NVidia drivers checking if they are running in a VM. NVidia sells cards specifically for running in a VM environment and attempts to block installation of drivers for consumer grade cards in a VM. You need to make sure to use the following in your domain.xml

<kvm>
 <hidden state='on'/>
</kvm>

See https://passthroughpo.st/apply-error-43-workaround/ and other resources for examples.

Here is a screenshot of my config:

QEMU GPU Passtrough Settings

Here is the "relevant" part of my domain.xml, I can share entire thing if you want, but it's got a bunch of unnecessary things.

  <os>
    <type arch='x86_64' machine='pc-i440fx-2.10'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/Windows10_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
  </features>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  • Unfortunately that did not work, I still have Windows in my QEMU window and a TianoCore splash on my passthrough card. Also, my card is recognized in device manager, but is stopped (error 43). – Stonecraft Nov 2 '18 at 1:06
  • 1
    Ok, so your GPU Passthrough is setup correctly as far as Linux/KVM is concerned, your issue now is the Code 43. This error means that NVidia drivers know they are running in a VM. It took me hours of trial and error to get it working w/ KVM where ESXi was much easier. I would recommend starting a thread on reddit.com/r/VFIO. If I remember correctly I had to use either UEFI/BIOS and/or Q35/I44X chipset but I forget the combo I used. I recommend using VIRSH and a known working domain.xml to define your host then edit as needed. – FreeSoftwareServers Nov 3 '18 at 1:37
  • I see you accepted, did you get it to work? You could post your own answer with what worked for you as well. After you could setup Virt-Manager in Windows if you follow my guide >> freesoftwareservers.com/display/FREES/… – FreeSoftwareServers Nov 3 '18 at 19:05
  • Not yet, but your answer got me through to the next problem, so I accepted. – Stonecraft Nov 4 '18 at 5:02
  • 1
    I use Fedora (RHEL Based) Distro since KVM is RHELs project, and yes I have found reasons to update to the latest qemu/kernel as fixes are still comming out often. KVM w/ GPU Passthrough has some issues like sound, but there are workarounds (like usb audio). When I changed architecture I think I was able to just re-use raw image and just delete and "recreate" vm via virt-manager but I could be wrong. – FreeSoftwareServers Nov 4 '18 at 22:10
3

I did it with this:

sudo qemu-system-x86_64 --enable-kvm \
-name TESTVM,process=TESTVM \
-drive id=disk0,if=virtio,cache=none,format=raw,file=/dev/nvme-tank/ntfs-zvol,index=2 \
-smp 4 \
-cpu core2duo,kvm=off \
-m 4096 \
-device vfio-pci,host=43:00.0,multifunction=on \
-device vfio-pci,host=43:00.1 \
-drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \
-drive if=pflash,format=raw,file=/usr/share/OVMF/OVMF_VARS.fd 

and then going to the GTX 1070 entry in Device Manager and updating the drivers. Last time (using virt-install) I installed updates from the NVIDIA website. Now to get this working with virt-install so that I can have an editable domain XML.

Edit: So far, when using virt-install, I now (again) get the Tiano screen, but this time installing the drivers as described above was not sufficient, nor did it work to import the above install using virt-install (on a side note, it took me a while to realize that all I had to do to make virt-install work with my existing installation was to include the boot iso, allowing it to boot from the installation without error).

Edit: So, I think I have it working now.

  1. Do the initial install using the above qemu-system-x86_64
  2. Updated drivers using Device Manager
  3. Then do virt-install (but get error 43 again):

    virt-install \ --name myWINVM1 \ --boot uefi \ --ram 4096 \ --features kvm_hidden=on \ --hostdev 43:00.0,address.type=pci,address.multifunction=on \ --hostdev 43:00.1,address.type=pci \ --disk path=/dev/nvme-tank/ntfs-zvol-part2 \ --cpu core2duo \ --vcpus 4 \ --os-type windows \ --os-variant win10 \ --network bridge=virbr0 \ --console pty,target_type=serial \ --disk /media/big-tank-8TB/OSISOS/Windows/WindowsOct2018.iso,device=cdrom \ --disk /media/big-tank-8TB/OSISOS/Windows/virtio-win-0.1.160.iso,device=cdrom

  4. Deleted the spice graphics device from the myWINVM1 XML. Still couldnt get past TianoCore.

  5. added the fake vendor ID line to the my XML

  6. Added my USB devices

my final xml:

<domain type='kvm' id='24'>
  <name>myWINVM1</name>
  <uuid>43e052b3-b3da-4025-92d8-ec7c8ff96ae9</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-bionic'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/myWINVM1_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='1234567890ab'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>core2duo</model>
    <feature policy='disable' name='ss'/>
    <feature policy='disable' name='monitor'/>
    <feature policy='require' name='cx16'/>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='lahf_lm'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/dev/nvme-tank/ntfs-zvol-part2'/>
      <backingStore/>
      <target dev='hda' bus='ide'/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/media/big-tank-8TB/OSISOS/Windows/WindowsOct2018.iso'/>
      <backingStore/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <alias name='ide0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/media/big-tank-8TB/OSISOS/Windows/virtio-win-0.1.160.iso'/>
      <backingStore/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:36:1d:c4'/>
      <source bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='rtl8139'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/4'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/4'>
      <source path='/dev/pts/4'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <sound model='ich6'>
      <alias name='sound0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc33b'/>
        <address bus='5' device='3'/>
      </source>
      <alias name='hostdev0'/>
      <address type='usb' bus='0' port='4'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc52b'/>
        <address bus='5' device='4'/>
      </source>
      <alias name='hostdev1'/>
      <address type='usb' bus='0' port='5'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x43' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x43' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir0'/>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir1'/>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='apparmor' relabel='yes'>
    <label>libvirt-43e052b3-b3da-4025-92d8-ec7c8ff96ae9</label>
    <imagelabel>libvirt-43e052b3-b3da-4025-92d8-ec7c8ff96ae9</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+64055:+120</label>
    <imagelabel>+64055:+120</imagelabel>
  </seclabel>
</domain>
  • I forgot to mention the "vendor_id", but I have seen that online, but I'm not sure it's needed. I would recommend using VIRTIO backed storage and virtio NIC. Adding VIRTIO storage is tricky, you'll want to add a second HDD w/ VIRTIO backing, get it working in Windows (by installing drivers) then switch primary HDD to VIRTIO and remove secondary HDD added just for drivers. The NIC isn't as hard, just add NIC w/ VirtIO and install drivers. They can all be found in the VirtIO ISO. Google will help :), Glad it's working! – FreeSoftwareServers Nov 9 '18 at 1:27
-1

This is what worked for me after a long struggle:

  • first of all edit the qemu VM configuration file this way:

     cd /etc/libvirt/qemu    
     sudo virsh define win10_GPU.xml
     sudo virsh edit win10_GPU
    

    where "win10_GPU" is your virtual machine name of course and add

     <qemu:commandline>
       <qemu:arg value='-cpu'/>
       <qemu:arg value='host,hv_time,kvm=off,hv_vendor_id=null,-hypervisor'/>   
     </qemu:commandline>
    

    after </devices> almost at the end of the file
    then hit ctrl+O (save), ctrl+X (exit) and type sudo virsh define win10_GPU.xml again;

  • remove every other display interface from your virtual machine before to even start install windows (like spice, video cirrus, whatever);

  • Do not install the 18xx build of windows10. I installed a 16xx version which is working perfectly even after the full updates.

This is my personal experience. Any other thing I tried led me to that error code 43.

  • It is false to say "Do not install 18xx Builds". If you see my comments above, I believe a different combination of chipset/bios is required when migrating from 16xx to 18xx. Eg, My Answer used 144x, but my comment says I re-created with v 1809 using q35 and UEFI. It's all about trial and error. Otherwise, you used qemu:commandline to do the same edit as "KVM HIDDEN OFF" in the domain xml. Either should work. – FreeSoftwareServers Jun 3 at 7:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.