Passthrough with vfio does not work : Debian freezes on boot

Getting your soundcard to work, using Debian on non-i386 hardware, etc

Passthrough with vfio does not work : Debian freezes on boot

Postby marietto » 2019-10-04 10:54

Hello.

I'm trying to enable the passthrough on my graphic card (nvidia RTX 2080 ti on CPU I9 - 9900k) on the debian 10,following this tutorial :


https://passthroughpo.st/gpu-debian/

These are the commands that I gave :


added this line : vfio_pci ids= 10de:1e04,10de:10f7 to this file : /etc/initramfs-tools/modules


below you can see how I have got that numbers :


ziomario@DESKTOP-N9UN2H3:/home/mariozio# lspci -nn | grep 01:00.

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] [10de:1e04] (rev a1)01:00.1 Audio device [0403]: NVIDIA Corporation TU102 High Definition Audio Controller [10de:10f7] (rev a1)


and then I did : update-initramfs -uupdate-initramfs -u -k all


and I have added these lines :


blacklist nouveau
options nouveau modeset-0


to this file : /etc/modprobe.d/blacklist.conf


the tutorial does not explain if I should modify also the file /etc/default/grub. Anyway I tried,adding this line :


GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on iommu=pt"


instead of the old one,that was : GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"


and then I did again : update-initramfs -uupdate-initramfs -u -k all


but the result does not change. During the boot I see the blinking cursor hanging on the screen and nothing happens anymore. I have even removed all the changes that I made and I did again :


update-initramfs -uupdate-initramfs -u -k all


to understand what's the problem and I seen that if I add this line : vfio_pci ids= 10de:1e04,10de:10f7


and then I reboots the machine,it freezes.


maybe is useful if u read here :

mariozio@DESKTOP-N9UN2H3:/home/mariozio# dmesg | grep -e DMAR

[ 0.006554] ACPI: DMAR 0x00000000393A0E98 0000A8 (v01 ALASKA A M I 00000002 01000013)
[ 0.163834] DMAR: IOMMU enabled
[ 0.222182] DMAR: Host address width 39
[ 0.222182] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.222185] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[ 0.222185] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.222187] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[ 0.222188] DMAR: RMRR base: 0x0000003988e000 end: 0x00000039ad7fff
[ 0.222188] DMAR: RMRR base: 0x0000003b000000 end: 0x0000003f7fffff
[ 0.222189] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.222189] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ 0.222190] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.223606] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 1.263105] DMAR: No ATSR found
[ 1.263138] DMAR: dmar0: Using Queued invalidation
[ 1.263179] DMAR: dmar1: Using Queued invalidation
[ 1.263426] DMAR: Hardware identity mapping for device 0000:00:00.0
[ 1.263427] DMAR: Hardware identity mapping for device 0000:00:01.0
[ 1.263469] DMAR: Hardware identity mapping for device 0000:00:02.0
[ 1.263470] DMAR: Hardware identity mapping for device 0000:00:12.0
[ 1.263471] DMAR: Hardware identity mapping for device 0000:00:14.0
[ 1.263471] DMAR: Hardware identity mapping for device 0000:00:14.2
[ 1.263472] DMAR: Hardware identity mapping for device 0000:00:16.0
[ 1.263472] DMAR: Hardware identity mapping for device 0000:00:17.0
[ 1.263473] DMAR: Hardware identity mapping for device 0000:00:1b.0
[ 1.263473] DMAR: Hardware identity mapping for device 0000:00:1c.0
[ 1.263474] DMAR: Hardware identity mapping for device 0000:00:1d.0
[ 1.263474] DMAR: Hardware identity mapping for device 0000:00:1f.0
[ 1.263475] DMAR: Hardware identity mapping for device 0000:00:1f.3
[ 1.263475] DMAR: Hardware identity mapping for device 0000:00:1f.4
[ 1.263476] DMAR: Hardware identity mapping for device 0000:00:1f.5
[ 1.263477] DMAR: Hardware identity mapping for device 0000:00:1f.6
[ 1.263478] DMAR: Hardware identity mapping for device 0000:01:00.0
[ 1.263479] DMAR: Hardware identity mapping for device 0000:01:00.1
[ 1.263480] DMAR: Hardware identity mapping for device 0000:01:00.2
[ 1.263480] DMAR: Hardware identity mapping for device 0000:01:00.3
[ 1.263481] DMAR: Setting RMRR:
[ 1.263481] DMAR: Ignoring identity map for HW passthrough device 0000:00:02.0 [0x3b000000 - 0x3f7fffff]
[ 1.263481] DMAR: Ignoring identity map for HW passthrough device 0000:00:14.0 [0x3988e000 - 0x39ad7fff]
[ 1.263482] DMAR: Prepare 0-16MiB unity mapping for LPC
[ 1.263482] DMAR: Ignoring identity map for HW passthrough device 0000:00:1f.0 [0x0 - 0xffffff]
[ 1.263529] DMAR: Intel(R) Virtualization Technology for Directed I/O
[ 16.610522] DMAR: 32bit 0000:01:00.0 uses non-identity mapping

Do u have some suggestion ? thanks.
marietto
 
Posts: 52
Joined: 2014-07-13 16:30

Re: Passthrough with vfio does not work : Debian freezes on

Postby CwF » 2019-10-05 12:58

Well, I disagree with the blog...

Nvidia has the most problems passing and results can vary.
Going from my own checklist;
Don't blacklist anything, not needed unless there is a bad driver
Code: Select all
#  lspci -nnk -d 10de:
04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107GL [Quadro K600] [10de:0ffa] (rev a1)
   Subsystem: Hewlett-Packard Company GK107GL [Quadro K600] [103c:094b]
   Kernel driver in use: vfio-pci
   Kernel modules: nouveau

notice how when things work, both are in use. Poorly written drivers don't hook and unhook under vfio control, nouveau does to the point the host can use it AND pass it to the vm AND pass it back on vm close. That's the theory at least...

Next, create /etc/modprobe.d/vfio.conf for you:
Code: Select all
options vfio_pci ids=10de:1e04,10de:10f7
#options vfio_iommu_type1 allow_unsafe_interrupts=0

the iommu type line is not always needed, the order of the ids doesn't matter, I have a few with 10+ devices.

Remove the entry from /etc/initramfs-tools/modules, yep nothing goes there...
You can have some modules listed there if otherwise not loaded, check and add if needed:
Code: Select all
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

I only use:
Code: Select all
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

I won't vouch for your RTX 2080 or your motherboard but I will state that most instructions out there miss one simple thing - the motherboard. The same solution does not always work on all boards. Slots matter, they are not all equal. If thing still don't work, mention the motherboard model. We might need to look to see the card is in a clean iommu group...etc...
CwF
 
Posts: 501
Joined: 2018-06-20 15:16

Re: Passthrough with vfio does not work : Debian freezes on

Postby marietto » 2019-10-05 15:20

mariozio@DESKTOP-N9UN2H3:/home/mariozio# lspci -knn | grep vfio

mariozio@DESKTOP-N9UN2H3:/home/mariozio# lspci -nnk -d 10de:

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] [10de:1e04] (rev a1)
Subsystem: ZOTAC International (MCO) Ltd. TU102 [GeForce RTX 2080 Ti] [19da:2503]
Kernel driver in use: nvidia
Kernel modules: nvidia
01:00.1 Audio device [0403]: NVIDIA Corporation TU102 High Definition Audio Controller [10de:10f7] (rev a1)
Subsystem: ZOTAC International (MCO) Ltd. TU102 High Definition Audio Controller [19da:2503]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
01:00.2 USB controller [0c03]: NVIDIA Corporation TU102 USB 3.1 Controller [10de:1ad6] (rev a1)
Subsystem: ZOTAC International (MCO) Ltd. TU102 USB 3.1 Controller [19da:2503]
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci
01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU102 UCSI Controller [10de:1ad7] (rev a1)
Subsystem: ZOTAC International (MCO) Ltd. TU102 UCSI Controller [19da:2503]

this output come from without the content below inside the file : /etc/initramfs-tools/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
marietto
 
Posts: 52
Joined: 2014-07-13 16:30

Re: Passthrough with vfio does not work : Debian freezes on

Postby CwF » 2019-10-05 15:41

marietto wrote:Kernel driver in use: nvidia
Kernel modules: nvidia


Remove
CwF
 
Posts: 501
Joined: 2018-06-20 15:16

Re: Passthrough with vfio does not work : Debian freezes on

Postby marietto » 2019-10-05 15:44

I added the lines below to the file : /etc/initramfs-tools/modules and the pc freezed on the boot stage.

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
marietto
 
Posts: 52
Joined: 2014-07-13 16:30

Re: Passthrough with vfio does not work : Debian freezes on

Postby marietto » 2019-10-05 15:45

what should I remove ? how ?
marietto
 
Posts: 52
Joined: 2014-07-13 16:30

Re: Passthrough with vfio does not work : Debian freezes on

Postby Head_on_a_Stick » 2019-10-05 16:00

Don't break DebianHow to report bugs

SharpBang GNU/Linux — a pre-configured Openbox/Tint2 desktop running on Debian stable
User avatar
Head_on_a_Stick
 
Posts: 10613
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: Passthrough with vfio does not work : Debian freezes on

Postby CwF » 2019-10-05 16:05

marietto wrote: /etc/initramfs-tools/modules and the pc freezed on the boot stage.

Clear it, only earlier kernels need them, as I said to check. The current buster kernel loads them all anyway.
edit that, some do need to be there..should not freeze the pc...

Remove the nvidia driver, however you remove it. I don't ever touch it. Clear any blacklist, nouveau can be installed if you'd like, doesn't matter but it will not conflict.

And what motherboard?
CwF
 
Posts: 501
Joined: 2018-06-20 15:16

Re: Passthrough with vfio does not work : Debian freezes on

Postby marietto » 2019-10-05 18:07

I'm trying to remove the nvidia driver from my system (running with debian 10). I tried several methods,but unsuccesfully. Something like these :

-) apt remove nvidia-driver
-) apt-get purge nvidia-driver

after that,when I reboot the PC,debian 10 restarts and I see this :

mario@DESKTOP-N9UN2H3:/home/mariozio# lspci -nnk -d 10de:
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] [10de:1e04] (rev a1)
Subsystem: ZOTAC International (MCO) Ltd. TU102 [GeForce RTX 2080 Ti] [19da:2503]
Kernel driver in use: nvidia
Kernel modules: nvidia

what to do ?

this is my MOBO :

http://it.gigabyte.com/products/page/mb ... rus_pro#kf
marietto
 
Posts: 52
Joined: 2014-07-13 16:30

Re: Passthrough with vfio does not work : Debian freezes on

Postby CwF » 2019-10-05 20:58

something like
Code: Select all
apt-get purge *nvidia*

ADD: make sure there are no xorg.conf files from running the nvidia install!
I assume you are using the intel video for the host system with a separate monitor and not trying to autostart a vm on a headless host on only one monitor.
If so, then the host needs no driver for the nvidia since it's not using it. As mentioned nouveau does not conflict with vfio. In this case the host should boot up on the intel with any error in the vm or vfio config. If something is preventing a clean boot on the intel, something else is wrong - I assume the presence of the proprietary nvidia driver. Once passed you may be able to use the nvidia driver, assuming you are passing it to a linux.

Next, I don't have much faith in the motherboard. From the manual, without a clear block diagram, mentions limitations in slot configuration. The #2x16 slot borrows 8 lanes from the #1x16 slot. I suspect like most of this type, ALL 16 lanes are in common with the Intel GPU. Without a clear diagram I do not know. Maybe a x16 slot gets 8 and the intel gets 8, maybe not.

To investigate, play with the below examples. Try to use the #2x16 slot instead of the #1x16 slot and see how it groups. Try the furthest x16(x4) slot the you'd hook to the chipset port and not the cpu port and it will likely work. I wouldn't worry about it being 'only' x4, it doesn't make much difference. This will heat up the chipset.
It's possible the first two slots shut off the intel gpu, some boards do. In the bios, the auto setting may need changed to force the intel on, so NOT auto.

To cut to the end of the story run in a root terminal:
Code: Select all
 for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done


Bad example:
Code: Select all
IOMMU group 1
[RESET]   00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 Processor Family Integrated Graphics Controller [8086:010a] (rev 09)
IOMMU group 1
   00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port [8086:0101] (rev 09)
   01:00.0 VGA compatible controller [0300]: NVIDIA Corporation G71GL [Quadro FX 1500] [10de:029e] (rev a1)

the above shows an integrated intel and then when the x16 slot is occupied. Not a clean iommu group and you get to play with ACS overrides and not my department!

A good example of a clean iommu group:
Code: Select all
IOMMU group 21
   04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107GL [Quadro K600] [10de:0ffa] (rev a1)
   04:00.1 Audio device [0403]: NVIDIA Corporation GK107 HDMI Audio Controller [10de:0e1b] (rev a1)

Good luck!
CwF
 
Posts: 501
Joined: 2018-06-20 15:16

Re: Passthrough with vfio does not work : Debian freezes on

Postby marietto » 2019-10-05 21:30

I have two monitors and two graphic boards. The first one is the nvidia rtx 2080 ti. the second one is embedded wth the mobo. Actually the monitors are attached both to the nvidia card. but in the future I want to use the nvidia card for the VM with windows and the internal graphic chipset for linux. This because I like to enjoy myself with the CG and Windows is better than Linux for this. But Linux is better than Windows for the protection against viruses and ransomwares.
marietto
 
Posts: 52
Joined: 2014-07-13 16:30

Re: Passthrough with vfio does not work : Debian freezes on

Postby marietto » 2019-10-05 21:47

I have enabled the internal graphic chipset on the BIOS and I have attached one monitor to the HDMI port of that. I've rebooted. On the boot messages I saw " failed to load kernel modules". And one only monitor is working (the one attached to the nvidia card).
marietto
 
Posts: 52
Joined: 2014-07-13 16:30

Re: Passthrough with vfio does not work : Debian freezes on

Postby CwF » 2019-10-06 00:44

marietto wrote:(the one attached to the nvidia card)

that's the critical part! This is a critical error.

If the goal is intel host on one monitor and a vm with the nvidia via vfio on monitor 2, then you need to back up. The host can NOT be configured to run the nvidia drivers and/or vfio, as far as your example suggest. I never install them, so I'm no help clearing that issue. You likely have an nvidia-detect created xorg.cong file, and the driver packages that all need removed. Or someone else knows how to make nvidia and vfio coexist. As said nouveau and vfio can coexist. Also an interesting side note, radeon can, amdgpu can not, so far in my experience.

Generally, a milestone list
1. Computer boots to a happy intel desktop
2. Install nvidia, and it doesn't care, still booting to the intel desktop!
3. Configure vfio, the vm, boot vm into second monitor...
4. wonder how to control it....
4 a. use virt-viewer or vmm-vm window to lock or passively grap kbm (best-full screen apps)
4 b. pass separate kbm (best-if 2 kbm's)
4 c. pass eth port and x2x or x2vnc host kbm into vm (best-windowed apps)
4 d. muck up iptables and firewalls with virt network rules
4 a+c if...
4 c if wayland is NOT used anywhere...etc

So at this point you need to re-achieve step 1.
kbm= keyboard and mouse
CwF
 
Posts: 501
Joined: 2018-06-20 15:16

Re: Passthrough with vfio does not work : Debian freezes on

Postby CwF » 2019-10-06 00:59

....also, if it were me I'd start with the nvidia in the furthest slot from the cpu (after the intel desktop is happy)

That should be a pci port served by the chipset (MCH, PCH, whatever) OR it is an unencumbered 4 lane port from the cpu, doubt it. In either case it should work and demonstrate the host config is correct. Then, at that success point, move the card to the preferable slot and try again. This involves NO change of the host, the vm would need the cards address corrected, or with vmm delete the device and re-attach...

I'm crossing my fingers, but typically that obvious 16x slot is not functional with the intel graphics active. Some E3 boards bifurcate to 8x intel and 8x slot. So if yours is 16/0 or 0/16 = sorry! Use the furthest slot...
CwF
 
Posts: 501
Joined: 2018-06-20 15:16

Re: Passthrough with vfio does not work : Debian freezes on

Postby marietto » 2019-10-06 08:30

For me it's ok if the host (OS = debian 10) uses the intel graphic chipset on one monitor and if the guest VM OS (Windows 10) uses the nvidia card passed with vfio. This because all the CG tools that I use runs better in Windows than in Linux,but Linux is better than Windows for the security. So,linux does not need to have the best graphic card that I have. Now,the problem is that only the nvidia card with the driver is working. The monitor attached with the intel graphic chipset is turned off. Should I fix this,before do something else ? Should I make work the intel graphic card with the monitor attached ? At this point I have no xorg.conf file on the debian machine.
marietto
 
Posts: 52
Joined: 2014-07-13 16:30

Next

Return to Hardware

Who is online

Users browsing this forum: No registered users and 7 guests

fashionable