Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

Problems with KVM+GPU passthrough

If none of the specific sub-forums seem right for your thread, ask here.
Post Reply
Message
Author
jen140
Posts: 15
Joined: 2009-10-03 20:46

Problems with KVM+GPU passthrough

#1 Post by jen140 »

Hello.

Please read everything before you answer. Thank you.
And yes, I've updated initramfs(update-initramfs -u), updated the grub config(grub-mkconfig -o /boot/grub/grub.cfg) and rebooted the machine multiple times.

I've been trying to setup KVM with GPU passthrough but I'm unable to do so.

Hardware:
Processor: Intel i7-7700K
Motherboard: Asus ex-8250-v7
GPU's: GeForce GTX 970, GeForce GTX 660

In the bios the only option that I have about vt-x/vt-d is "Intel VT-x Thchnology" that is Supported.

Software:
Debian 9
qemu-kvm 1:2.8

What I want to do:
I'd like to have this machine running with kvm and two windows 10 VM's, both having each one of the GPU's. I'm not interested in running any desktop manager on the host(and I don't have xorg installed).

The problem:
virt-manager says "unsupported configuration: host doesn't support passthrough of host PCI devices".

/etc/defaults/grub:

Code: Select all

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio-pci.ids=10de:13c2,10de:0fbb,10de:11c0,10de:0e0b"
GRUB_CMDLINE_LINUX=""
/etc/modules:

Code: Select all

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
/etc/modprobe.d/nouveau.conf:

Code: Select all

blacklist nouveau
/etc/modprobe.d/vfio.conf:

Code: Select all

options vfio-pci ids=10de:13c2,10de:0fbb,10de:11c0,10de:0e0b
/etc/initramfs-tools/modules:

Code: Select all

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
egrep -q '^flags.*(svm|vmx)' /proc/cpuinfo && echo virtualization extensions available:

Code: Select all

virtualization extensions available
ls -liah /sys/kernel/iommu_groups/:

Code: Select all

total 0
2708 drwxr-xr-x  2 root root 0 Dec 22 14:09 .
2676 drwxr-xr-x 10 root root 0 Dec 22 13:14 ..
lspci -nn:

Code: Select all

00:00.0 Host bridge [0600]: Intel Corporation Device [8086:591f] (rev 05)
00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 05)
00:14.0 USB controller [0c03]: Intel Corporation Device [8086:a2af]
00:16.0 Communication controller [0780]: Intel Corporation Device [8086:a2ba]
00:17.0 SATA controller [0106]: Intel Corporation Device [8086:a282]
00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:a294] (rev f0)
00:1d.0 PCI bridge [0604]: Intel Corporation Device [8086:a298] (rev f0)
00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a2c8]
00:1f.2 Memory controller [0580]: Intel Corporation Device [8086:a2a1]
00:1f.3 Audio device [0403]: Intel Corporation Device [8086:a2f0]
00:1f.4 SMBus [0c05]: Intel Corporation Device [8086:a2a3]
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I219-V [8086:15b8]
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GM204 High Definition Audio Controller [10de:0fbb] (rev a1)
03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK106 [GeForce GTX 660] [10de:11c0] (rev a1)
03:00.1 Audio device [0403]: NVIDIA Corporation GK106 HDMI Audio Controller [10de:0e0b] (rev a1)
lspci -k:

Code: Select all

00:00.0 Host bridge: Intel Corporation Device 591f (rev 05)
	Subsystem: ASUSTeK Computer Inc. Device 8694
00:01.0 PCI bridge: Intel Corporation Skylake PCIe Controller (x16) (rev 05)
	Kernel driver in use: pcieport
	Kernel modules: shpchp
00:14.0 USB controller: Intel Corporation Device a2af
	Subsystem: ASUSTeK Computer Inc. Device 8694
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci
00:16.0 Communication controller: Intel Corporation Device a2ba
	Subsystem: ASUSTeK Computer Inc. Device 8694
	Kernel driver in use: mei_me
	Kernel modules: mei_me
00:17.0 SATA controller: Intel Corporation Device a282
	Subsystem: ASUSTeK Computer Inc. Device 8694
	Kernel driver in use: ahci
	Kernel modules: ahci
00:1c.0 PCI bridge: Intel Corporation Device a294 (rev f0)
	Kernel driver in use: pcieport
	Kernel modules: shpchp
00:1d.0 PCI bridge: Intel Corporation Device a298 (rev f0)
	Kernel driver in use: pcieport
	Kernel modules: shpchp
00:1f.0 ISA bridge: Intel Corporation Device a2c8
	Subsystem: ASUSTeK Computer Inc. Device 8694
00:1f.2 Memory controller: Intel Corporation Device a2a1
	Subsystem: ASUSTeK Computer Inc. Device 8694
00:1f.3 Audio device: Intel Corporation Device a2f0
	Subsystem: ASUSTeK Computer Inc. Device 86ee
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
00:1f.4 SMBus: Intel Corporation Device a2a3
	Subsystem: ASUSTeK Computer Inc. Device 8694
	Kernel driver in use: i801_smbus
	Kernel modules: i2c_i801
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (2) I219-V
	Subsystem: ASUSTeK Computer Inc. Ethernet Connection (2) I219-V
	Kernel driver in use: e1000e
	Kernel modules: e1000e
01:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 970] (rev a1)
	Subsystem: Micro-Star International Co., Ltd. [MSI] GM204 [GeForce GTX 970]
	Kernel modules: nouveau
01:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1)
	Subsystem: Micro-Star International Co., Ltd. [MSI] GM204 High Definition Audio Controller
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
03:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 660] (rev a1)
	Subsystem: Micro-Star International Co., Ltd. [MSI] GK106 [GeForce GTX 660]
	Kernel modules: nouveau
03:00.1 Audio device: NVIDIA Corporation GK106 HDMI Audio Controller (rev a1)
	Subsystem: Micro-Star International Co., Ltd. [MSI] GK106 HDMI Audio Controller
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
dmesg |grep -i vfio:

Code: Select all

[    1.862810] VFIO - User Level meta-driver version: 0.3
[    1.870620] vfio-pci: probe of 0000:01:00.0 failed with error -22
[    1.870627] vfio_pci: add [10de:13c2[ffff:ffff]] class 0x000000/00000000
[    1.870642] vfio-pci: probe of 0000:01:00.0 failed with error -22
[    1.870652] vfio-pci: probe of 0000:01:00.1 failed with error -22
[    1.870655] vfio_pci: add [10de:0fbb[ffff:ffff]] class 0x000000/00000000
[    1.870666] vfio-pci: probe of 0000:01:00.0 failed with error -22
[    1.870675] vfio-pci: probe of 0000:01:00.1 failed with error -22
[    1.870685] vfio-pci: probe of 0000:03:00.0 failed with error -22
[    1.870687] vfio_pci: add [10de:11c0[ffff:ffff]] class 0x000000/00000000
[    1.870699] vfio-pci: probe of 0000:01:00.0 failed with error -22
[    1.870707] vfio-pci: probe of 0000:01:00.1 failed with error -22
[    1.870716] vfio-pci: probe of 0000:03:00.0 failed with error -22
[    1.870726] vfio-pci: probe of 0000:03:00.1 failed with error -22
[    1.870729] vfio_pci: add [10de:0e0b[ffff:ffff]] class 0x000000/00000000
dmesg | grep -e DMAR -e IOMMU:

Code: Select all

[    0.000000] DMAR: IOMMU enabled
[    0.477982] AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[    0.477982] AMD IOMMUv2 functionality not available on this system

driver/unbind:

Code: Select all

echo '0000:01:00.0' | /sys/bus/pci/devices/0000\:01\:00.0/driver/unbind:
bash: /sys/bus/pci/devices/0000:01:00.0/driver/unbind: No such file or directory

echo '0000:01:00.1' | /sys/bus/pci/devices/0000\:01\:00.1/driver/unbind:
bash: /sys/bus/pci/devices/0000:01:00.1/driver/unbind: Permission denied

echo '0000:03:00.0' | /sys/bus/pci/devices/0000\:03\:00.0/driver/unbind
bash: /sys/bus/pci/devices/0000:03:00.0/driver/unbind: No such file or directory

echo '0000:03:00.1' | /sys/bus/pci/devices/0000\:03\:00.1/driver/unbind
bash: /sys/bus/pci/devices/0000:03:00.1/driver/unbind: Permission denied
lsmod | grep kvm:

Code: Select all

kvm_intel             192512  0
kvm                   589824  1 kvm_intel
irqbypass              16384  2 kvm,vfio_pci
lsmod | grep vfio:

Code: Select all

vfio_pci               45056  0
irqbypass              16384  2 kvm,vfio_pci
vfio_virqfd            16384  1 vfio_pci
vfio_iommu_type1       20480  0
vfio                   28672  2 vfio_iommu_type1,vfio_pci
Full dmesg:
https://paste.ee/p/Osq7K

Thanks for your attention.

Post Reply