KVM/VGA Passthrough - Blackscreen on host & guest

Kernels & Hardware, configuring network, installing services

KVM/VGA Passthrough - Blackscreen on host & guest

Postby Domme » 2018-01-28 14:41

Hello,

my new notebook doesn't support Windows 7 anymore so I decided to set up a Guest Windows 7 VM under Host Debian 9.3 (4.9.0.5) with VGA Passthrough to avoid the pain of using Windows 10. I went through many many different tutorials and tested many approaches but still couldn't manage to make it work. It has been a few weeks now and I'm starting to question my ability to solve this problem alone so I decided to seek some help and hope I will find the missing piece to finally finish my setup.

Hardware
  • CPU: i7-7700 (yes, the desktop CPU)
  • GPU: GTX 1060 (mobile 6GB version)
  • Chipset: Z170 (VT enabled in BIOS)

First of all, I modified /etc/default/grub to enable IOMMU and ran update-grub.

    [...]
    GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt"
    [...]

After a reboot, the IOMMU groups are initiated:

Code: Select all
$ sudo dmesg | grep -e IOMMU -e DMAR

[    0.000000] ACPI: DMAR 0x000000002EBAD968 000070 (v01 ALASKA A M I    00000001 INTL 00000001)
[    0.000000] DMAR: IOMMU enabled
[    0.042405] DMAR: Host address width 39
[    0.042407] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[    0.042412] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.042415] DMAR: RMRR base: 0x0000002e5c8000 end: 0x0000002e5e7fff
[    0.042418] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed90000 IOMMU 0
[    0.042420] DMAR-IR: HPET id 0 under DRHD base 0xfed90000
[    0.042421] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.043687] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.557424] DMAR: No ATSR found
[    0.557745] DMAR: dmar0: Using Queued invalidation
[    0.557791] DMAR: Hardware identity mapping for device 0000:00:00.0
[    0.557793] DMAR: Hardware identity mapping for device 0000:00:01.0
[    0.557795] DMAR: Hardware identity mapping for device 0000:00:08.0
[    0.557797] DMAR: Hardware identity mapping for device 0000:00:14.0
[    0.557799] DMAR: Hardware identity mapping for device 0000:00:14.2
[    0.557801] DMAR: Hardware identity mapping for device 0000:00:16.0
[    0.557803] DMAR: Hardware identity mapping for device 0000:00:17.0
[    0.557805] DMAR: Hardware identity mapping for device 0000:00:1c.0
[    0.557807] DMAR: Hardware identity mapping for device 0000:00:1c.6
[    0.557809] DMAR: Hardware identity mapping for device 0000:00:1c.7
[    0.557811] DMAR: Hardware identity mapping for device 0000:00:1d.0
[    0.557813] DMAR: Hardware identity mapping for device 0000:00:1d.4
[    0.557815] DMAR: Hardware identity mapping for device 0000:00:1f.0
[    0.557818] DMAR: Hardware identity mapping for device 0000:00:1f.2
[    0.557820] DMAR: Hardware identity mapping for device 0000:00:1f.3
[    0.557822] DMAR: Hardware identity mapping for device 0000:00:1f.4
[    0.557825] DMAR: Hardware identity mapping for device 0000:01:00.0
[    0.557827] DMAR: Hardware identity mapping for device 0000:01:00.1
[    0.557830] DMAR: Hardware identity mapping for device 0000:6d:00.0
[    0.557833] DMAR: Hardware identity mapping for device 0000:6e:00.0
[    0.557836] DMAR: Hardware identity mapping for device 0000:70:00.0
[    0.557838] DMAR: Setting RMRR:
[    0.557840] DMAR: Ignoring identity map for HW passthrough device 0000:00:14.0 [0x2e5c8000 - 0x2e5e7fff]
[    0.557843] DMAR: Prepare 0-16MiB unity mapping for LPC
[    0.557845] DMAR: Ignoring identity map for HW passthrough device 0000:00:1f.0 [0x0 - 0xffffff]
[    0.557848] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    1.143547] AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[    1.143561] AMD IOMMUv2 functionality not available on this system
[    2.212461] DMAR: 32bit 0000:6e:00.0 uses non-identity mapping


My dedicated GPU is located in IOMMU group 1:

Code: Select all
$ for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo "IOMMU group $(basename "$iommu_group")"; for device in $(ls -1 "$iommu_group"/devices/); do echo -n $'\t'; lspci -nns "$device"; done; done

00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 05)
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106M [GeForce GTX 1060] [10de:1c60] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f1] (rev a1)


I edited /etc/modules to contain the necessary modules.

Code: Select all
pci_stub
vfio
vfio_iommu_type1
vfio_pci
kvm
kvm_intel


And assigned the GPU (+soundcard) PCI IDs to pci_stub in /etc/initramfs-tools/modules.

Note: vfio_pci tends to leave my system unbootable with intel_iommu=on in the cmdline, that's why I use pci-stub.

Code: Select all
pci_stub ids=10de:1c60,10de:10f1


The devices are claimed by pci-stub:

Code: Select all
$ sudo  dmesg | grep pci-stub

[    3.846977] pci-stub: add 10DE:1C60 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    3.847682] pci-stub 0000:01:00.0: claimed by stub
[    3.848385] pci-stub: add 10DE:10F1 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    3.849088] pci-stub 0000:01:00.1: claimed by stub

$ lspci -nnk

[...]
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106M [GeForce GTX 1060] [10de:1c60] (rev a1)
   Subsystem: CLEVO/KAPOK Computer GP106M [GeForce GTX 1060] [1558:7507]
   Kernel driver in use: pci-stub
   Kernel modules: nouveau
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f1] (rev a1)
   Subsystem: CLEVO/KAPOK Computer Device [1558:7507]
   Kernel driver in use: pci-stub
   Kernel modules: snd_hda_intel
[...]


However, when I run the VM inside virt-manager the host system's screen turns black after a short delay of <1s. There's no output on the guest monitor connected via HDMI, either. The music on the host keeps playing and SSH still works. Before adding the GPU as PCI Host device the VM boots fine, in a window of course, but without screwing up the host's display. Could my notebook have a problem with sending the video signal to the correct display? How does this working in general, especially on a notebook? Could the problem be associated with the GPU and audio device being listed as subsystem? Any help is greatly appreciated!


Best regards,
Domme
Last edited by Domme on 2018-02-09 01:21, edited 1 time in total.
Domme
 
Posts: 2
Joined: 2018-01-28 08:16

Re: KVM/VGA Passthrough - Blackscreen on host & guest

Postby te36 » 2018-01-29 18:13

You did not mention what notebook you are using.

Notebooks with a discrete GPU like your GTX 1060 have whats called hybrid graphics. At least the notebooks display is connected to both the onboard GPU (Intel HD 630 on your i7700) and the discrete GPU. But i do not know how the display is switched. This article mentions that some notebooks have additional internal HW (video switch) to do this:

viewtopic.php?f=5&t=136286

If i where you i would first try to figure out how debian/linux deals with this switching in the absence of any virtual machines.

If you want to use PCI passthrough to view Windows on your notebook screen with PCI passthrough, i think you need to be able to

a) make debian ignore that GPU, aka: via vfio/pci_stub
b) be able to explicitly switch display to discrete graphics or have it switch automatically to discrete graphics when there is a video signal coming from it (from your windows driver in the VM).

If you are lucky, then what you experience is just the VM starting, the GPU gets initialized enough so that the notebook screen is switched to it - and then your VM setup is incorrect and the VM hangs.

I would guess that you did not set up the VM with UEFI/Q35 chipset. The GTX 1060 will not boot with legacy BIOS, that would give the immediate black screen i think.

Not sure if its worth going through the trouble of PCI passthrough just for the native 3D acceleration. Did you try VirtualBox to check how good the 3D acceleration without PCI passthrough via the linux driver is ?

Not sure why you want to use Windows 7. For any new games, Windows 10 should be prefferrable.

I just managed to get PCI passthrough on a desktop debian PC with GTX 1050 ti to work, after fighting with the scatter-gather documentation on various web pages. Now i have to see that all the 3D features i want will work: HDMI-1.4 3D to my projector (Nvidia 3DTV) and 144Hz G-SYNC to a desktop 3D display. Thats why i am forced to use PCI passthrough because Linux does not support either *sigh*
te36
 
Posts: 4
Joined: 2018-01-29 17:36

Re: KVM/VGA Passthrough - Blackscreen on host & guest

Postby Domme » 2018-02-09 01:47

Thanks for your reply. I finally found some time to continue my journey and did some more research.

te36 wrote:You did not mention what notebook you are using.

It's a Schenker DTR15 aka XMG U507, based on Clevo P7xxDM2(-G).

te36 wrote:Notebooks with a discrete GPU like your GTX 1060 have whats called hybrid graphics. At least the notebooks display is connected to both the onboard GPU (Intel HD 630 on your i7700) and the discrete GPU. But i do not know how the display is switched. This article mentions that some notebooks have additional internal HW (video switch) to do this:

viewtopic.php?f=5&t=136286

The vendor's page says "The notebook doesn't use NVIDIA Optimus. Both internal and external image processing is done by the NVIDIA dGPU. BTW The link you mentioned is redirecting to this thread.

te36 wrote:make debian ignore that GPU, aka: via vfio/pci_stub

Seems to work when I boot with vfio-pci (at least I get a display freeze, hooray!). I will try again if I can SSH into it.

te36 wrote:be able to explicitly switch display to discrete graphics or have it switch automatically to discrete graphics when there is a video signal coming from it (from your windows driver in the VM).

Seems to be obsolete now because effectively I only have one GPU

te36 wrote:If you are lucky, then what you experience is just the VM starting, the GPU gets initialized enough so that the notebook screen is switched to it - and then your VM setup is incorrect and the VM hangs.

I think I'm lucky then because the switch takes place but nothing happens afterwards. Maybe I need to shutdown X properly to make it work.

te36 wrote:I would guess that you did not set up the VM with UEFI/Q35 chipset. The GTX 1060 will not boot with legacy BIOS, that would give the immediate black screen i think.

It is set to UEFI/i440FX. Will try Q35 once I have more time.

te36 wrote:Not sure if its worth going through the trouble of PCI passthrough just for the native 3D acceleration. Did you try VirtualBox to check how good the 3D acceleration without PCI passthrough via the linux driver is ?

Didn't try that because I want to strive to the maximum, not make any compromise, even if it takes longer. It's more of a philosophical thing...

te36 wrote:Not sure why you want to use Windows 7. For any new games, Windows 10 should be prefferrable.

I know, this is also more of a philosophical thing. I want to avoid the privacy breach of Windows 10. If that's not possible, I will lock Win10 on it's own virtual harddrive but never directly boot Win10 because I don't want it to meddle with my other partitions. That has happened before, so I set everything up anew, only to see Win10 brick itself after less than a week... I was a bit pissed and don't trust this OS anymore.

te36 wrote:I just managed to get PCI passthrough on a desktop debian PC with GTX 1050 ti to work, after fighting with the scatter-gather documentation on various web pages. Now i have to see that all the 3D features i want will work: HDMI-1.4 3D to my projector (Nvidia 3DTV) and 144Hz G-SYNC to a desktop 3D display. Thats why i am forced to use PCI passthrough because Linux does not support either *sigh*

Congratulations! If you can provide some links I would be happy, but I think I just have to read through the tutorials I already went through a few more times to fully understand the concept =)
Domme
 
Posts: 2
Joined: 2018-01-28 08:16


Return to System configuration

Who is online

Users browsing this forum: No registered users and 4 guests

fashionable