Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

Passthrough with vfio does not work : Debian freezes on boot

Need help with peripherals or devices?
Message
Author
marietto
Posts: 80
Joined: 2014-07-13 16:30

Passthrough with vfio does not work : Debian freezes on boot

#1 Post by marietto »

Hello.

I'm trying to enable the passthrough on my graphic card (nvidia RTX 2080 ti on CPU I9 - 9900k) on the debian 10,following this tutorial :


https://passthroughpo.st/gpu-debian/

These are the commands that I gave :


added this line : vfio_pci ids= 10de:1e04,10de:10f7 to this file : /etc/initramfs-tools/modules


below you can see how I have got that numbers :


ziomario@DESKTOP-N9UN2H3:/home/mariozio# lspci -nn | grep 01:00.

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] [10de:1e04] (rev a1)01:00.1 Audio device [0403]: NVIDIA Corporation TU102 High Definition Audio Controller [10de:10f7] (rev a1)


and then I did : update-initramfs -uupdate-initramfs -u -k all


and I have added these lines :


blacklist nouveau
options nouveau modeset-0


to this file : /etc/modprobe.d/blacklist.conf


the tutorial does not explain if I should modify also the file /etc/default/grub. Anyway I tried,adding this line :


GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on iommu=pt"


instead of the old one,that was : GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"


and then I did again : update-initramfs -uupdate-initramfs -u -k all


but the result does not change. During the boot I see the blinking cursor hanging on the screen and nothing happens anymore. I have even removed all the changes that I made and I did again :


update-initramfs -uupdate-initramfs -u -k all


to understand what's the problem and I seen that if I add this line : vfio_pci ids= 10de:1e04,10de:10f7


and then I reboots the machine,it freezes.


maybe is useful if u read here :

mariozio@DESKTOP-N9UN2H3:/home/mariozio# dmesg | grep -e DMAR

[ 0.006554] ACPI: DMAR 0x00000000393A0E98 0000A8 (v01 ALASKA A M I 00000002 01000013)
[ 0.163834] DMAR: IOMMU enabled
[ 0.222182] DMAR: Host address width 39
[ 0.222182] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.222185] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[ 0.222185] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.222187] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[ 0.222188] DMAR: RMRR base: 0x0000003988e000 end: 0x00000039ad7fff
[ 0.222188] DMAR: RMRR base: 0x0000003b000000 end: 0x0000003f7fffff
[ 0.222189] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.222189] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ 0.222190] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.223606] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 1.263105] DMAR: No ATSR found
[ 1.263138] DMAR: dmar0: Using Queued invalidation
[ 1.263179] DMAR: dmar1: Using Queued invalidation
[ 1.263426] DMAR: Hardware identity mapping for device 0000:00:00.0
[ 1.263427] DMAR: Hardware identity mapping for device 0000:00:01.0
[ 1.263469] DMAR: Hardware identity mapping for device 0000:00:02.0
[ 1.263470] DMAR: Hardware identity mapping for device 0000:00:12.0
[ 1.263471] DMAR: Hardware identity mapping for device 0000:00:14.0
[ 1.263471] DMAR: Hardware identity mapping for device 0000:00:14.2
[ 1.263472] DMAR: Hardware identity mapping for device 0000:00:16.0
[ 1.263472] DMAR: Hardware identity mapping for device 0000:00:17.0
[ 1.263473] DMAR: Hardware identity mapping for device 0000:00:1b.0
[ 1.263473] DMAR: Hardware identity mapping for device 0000:00:1c.0
[ 1.263474] DMAR: Hardware identity mapping for device 0000:00:1d.0
[ 1.263474] DMAR: Hardware identity mapping for device 0000:00:1f.0
[ 1.263475] DMAR: Hardware identity mapping for device 0000:00:1f.3
[ 1.263475] DMAR: Hardware identity mapping for device 0000:00:1f.4
[ 1.263476] DMAR: Hardware identity mapping for device 0000:00:1f.5
[ 1.263477] DMAR: Hardware identity mapping for device 0000:00:1f.6
[ 1.263478] DMAR: Hardware identity mapping for device 0000:01:00.0
[ 1.263479] DMAR: Hardware identity mapping for device 0000:01:00.1
[ 1.263480] DMAR: Hardware identity mapping for device 0000:01:00.2
[ 1.263480] DMAR: Hardware identity mapping for device 0000:01:00.3
[ 1.263481] DMAR: Setting RMRR:
[ 1.263481] DMAR: Ignoring identity map for HW passthrough device 0000:00:02.0 [0x3b000000 - 0x3f7fffff]
[ 1.263481] DMAR: Ignoring identity map for HW passthrough device 0000:00:14.0 [0x3988e000 - 0x39ad7fff]
[ 1.263482] DMAR: Prepare 0-16MiB unity mapping for LPC
[ 1.263482] DMAR: Ignoring identity map for HW passthrough device 0000:00:1f.0 [0x0 - 0xffffff]
[ 1.263529] DMAR: Intel(R) Virtualization Technology for Directed I/O
[ 16.610522] DMAR: 32bit 0000:01:00.0 uses non-identity mapping

Do u have some suggestion ? thanks.

CwF
Global Moderator
Global Moderator
Posts: 2638
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 192 times

Re: Passthrough with vfio does not work : Debian freezes on

#2 Post by CwF »

Well, I disagree with the blog...

Nvidia has the most problems passing and results can vary.
Going from my own checklist;
Don't blacklist anything, not needed unless there is a bad driver

Code: Select all

#  lspci -nnk -d 10de:
04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107GL [Quadro K600] [10de:0ffa] (rev a1)
	Subsystem: Hewlett-Packard Company GK107GL [Quadro K600] [103c:094b]
	Kernel driver in use: vfio-pci
	Kernel modules: nouveau
notice how when things work, both are in use. Poorly written drivers don't hook and unhook under vfio control, nouveau does to the point the host can use it AND pass it to the vm AND pass it back on vm close. That's the theory at least...

Next, create /etc/modprobe.d/vfio.conf for you:

Code: Select all

options vfio_pci ids=10de:1e04,10de:10f7
#options vfio_iommu_type1 allow_unsafe_interrupts=0
the iommu type line is not always needed, the order of the ids doesn't matter, I have a few with 10+ devices.

Remove the entry from /etc/initramfs-tools/modules, yep nothing goes there...
You can have some modules listed there if otherwise not loaded, check and add if needed:

Code: Select all

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
I only use:

Code: Select all

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
I won't vouch for your RTX 2080 or your motherboard but I will state that most instructions out there miss one simple thing - the motherboard. The same solution does not always work on all boards. Slots matter, they are not all equal. If thing still don't work, mention the motherboard model. We might need to look to see the card is in a clean iommu group...etc...

marietto
Posts: 80
Joined: 2014-07-13 16:30

Re: Passthrough with vfio does not work : Debian freezes on

#3 Post by marietto »

mariozio@DESKTOP-N9UN2H3:/home/mariozio# lspci -knn | grep vfio

mariozio@DESKTOP-N9UN2H3:/home/mariozio# lspci -nnk -d 10de:

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] [10de:1e04] (rev a1)
Subsystem: ZOTAC International (MCO) Ltd. TU102 [GeForce RTX 2080 Ti] [19da:2503]
Kernel driver in use: nvidia
Kernel modules: nvidia
01:00.1 Audio device [0403]: NVIDIA Corporation TU102 High Definition Audio Controller [10de:10f7] (rev a1)
Subsystem: ZOTAC International (MCO) Ltd. TU102 High Definition Audio Controller [19da:2503]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
01:00.2 USB controller [0c03]: NVIDIA Corporation TU102 USB 3.1 Controller [10de:1ad6] (rev a1)
Subsystem: ZOTAC International (MCO) Ltd. TU102 USB 3.1 Controller [19da:2503]
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci
01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU102 UCSI Controller [10de:1ad7] (rev a1)
Subsystem: ZOTAC International (MCO) Ltd. TU102 UCSI Controller [19da:2503]

this output come from without the content below inside the file : /etc/initramfs-tools/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

CwF
Global Moderator
Global Moderator
Posts: 2638
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 192 times

Re: Passthrough with vfio does not work : Debian freezes on

#4 Post by CwF »

marietto wrote:Kernel driver in use: nvidia
Kernel modules: nvidia
Remove

marietto
Posts: 80
Joined: 2014-07-13 16:30

Re: Passthrough with vfio does not work : Debian freezes on

#5 Post by marietto »

I added the lines below to the file : /etc/initramfs-tools/modules and the pc freezed on the boot stage.

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

marietto
Posts: 80
Joined: 2014-07-13 16:30

Re: Passthrough with vfio does not work : Debian freezes on

#6 Post by marietto »

what should I remove ? how ?

User avatar
Head_on_a_Stick
Posts: 14114
Joined: 2014-06-01 17:46
Location: London, England
Has thanked: 81 times
Been thanked: 132 times

Re: Passthrough with vfio does not work : Debian freezes on

#7 Post by Head_on_a_Stick »

deadbang

CwF
Global Moderator
Global Moderator
Posts: 2638
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 192 times

Re: Passthrough with vfio does not work : Debian freezes on

#8 Post by CwF »

marietto wrote: /etc/initramfs-tools/modules and the pc freezed on the boot stage.
Clear it, only earlier kernels need them, as I said to check. The current buster kernel loads them all anyway.
edit that, some do need to be there..should not freeze the pc...

Remove the nvidia driver, however you remove it. I don't ever touch it. Clear any blacklist, nouveau can be installed if you'd like, doesn't matter but it will not conflict.

And what motherboard?

marietto
Posts: 80
Joined: 2014-07-13 16:30

Re: Passthrough with vfio does not work : Debian freezes on

#9 Post by marietto »

I'm trying to remove the nvidia driver from my system (running with debian 10). I tried several methods,but unsuccesfully. Something like these :

-) apt remove nvidia-driver
-) apt-get purge nvidia-driver

after that,when I reboot the PC,debian 10 restarts and I see this :

mario@DESKTOP-N9UN2H3:/home/mariozio# lspci -nnk -d 10de:
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] [10de:1e04] (rev a1)
Subsystem: ZOTAC International (MCO) Ltd. TU102 [GeForce RTX 2080 Ti] [19da:2503]
Kernel driver in use: nvidia
Kernel modules: nvidia

what to do ?

this is my MOBO :

http://it.gigabyte.com/products/page/mb ... rus_pro#kf

CwF
Global Moderator
Global Moderator
Posts: 2638
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 192 times

Re: Passthrough with vfio does not work : Debian freezes on

#10 Post by CwF »

something like

Code: Select all

apt-get purge *nvidia*
ADD: make sure there are no xorg.conf files from running the nvidia install!
I assume you are using the intel video for the host system with a separate monitor and not trying to autostart a vm on a headless host on only one monitor.
If so, then the host needs no driver for the nvidia since it's not using it. As mentioned nouveau does not conflict with vfio. In this case the host should boot up on the intel with any error in the vm or vfio config. If something is preventing a clean boot on the intel, something else is wrong - I assume the presence of the proprietary nvidia driver. Once passed you may be able to use the nvidia driver, assuming you are passing it to a linux.

Next, I don't have much faith in the motherboard. From the manual, without a clear block diagram, mentions limitations in slot configuration. The #2x16 slot borrows 8 lanes from the #1x16 slot. I suspect like most of this type, ALL 16 lanes are in common with the Intel GPU. Without a clear diagram I do not know. Maybe a x16 slot gets 8 and the intel gets 8, maybe not.

To investigate, play with the below examples. Try to use the #2x16 slot instead of the #1x16 slot and see how it groups. Try the furthest x16(x4) slot the you'd hook to the chipset port and not the cpu port and it will likely work. I wouldn't worry about it being 'only' x4, it doesn't make much difference. This will heat up the chipset.
It's possible the first two slots shut off the intel gpu, some boards do. In the bios, the auto setting may need changed to force the intel on, so NOT auto.

To cut to the end of the story run in a root terminal:

Code: Select all

 for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done
Bad example:

Code: Select all

IOMMU group 1
[RESET]	00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 Processor Family Integrated Graphics Controller [8086:010a] (rev 09)
IOMMU group 1
	00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port [8086:0101] (rev 09)
	01:00.0 VGA compatible controller [0300]: NVIDIA Corporation G71GL [Quadro FX 1500] [10de:029e] (rev a1)
the above shows an integrated intel and then when the x16 slot is occupied. Not a clean iommu group and you get to play with ACS overrides and not my department!

A good example of a clean iommu group:

Code: Select all

IOMMU group 21
	04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107GL [Quadro K600] [10de:0ffa] (rev a1)
	04:00.1 Audio device [0403]: NVIDIA Corporation GK107 HDMI Audio Controller [10de:0e1b] (rev a1)
Good luck!

marietto
Posts: 80
Joined: 2014-07-13 16:30

Re: Passthrough with vfio does not work : Debian freezes on

#11 Post by marietto »

I have two monitors and two graphic boards. The first one is the nvidia rtx 2080 ti. the second one is embedded wth the mobo. Actually the monitors are attached both to the nvidia card. but in the future I want to use the nvidia card for the VM with windows and the internal graphic chipset for linux. This because I like to enjoy myself with the CG and Windows is better than Linux for this. But Linux is better than Windows for the protection against viruses and ransomwares.

marietto
Posts: 80
Joined: 2014-07-13 16:30

Re: Passthrough with vfio does not work : Debian freezes on

#12 Post by marietto »

I have enabled the internal graphic chipset on the BIOS and I have attached one monitor to the HDMI port of that. I've rebooted. On the boot messages I saw " failed to load kernel modules". And one only monitor is working (the one attached to the nvidia card).

CwF
Global Moderator
Global Moderator
Posts: 2638
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 192 times

Re: Passthrough with vfio does not work : Debian freezes on

#13 Post by CwF »

marietto wrote:(the one attached to the nvidia card)
that's the critical part! This is a critical error.

If the goal is intel host on one monitor and a vm with the nvidia via vfio on monitor 2, then you need to back up. The host can NOT be configured to run the nvidia drivers and/or vfio, as far as your example suggest. I never install them, so I'm no help clearing that issue. You likely have an nvidia-detect created xorg.cong file, and the driver packages that all need removed. Or someone else knows how to make nvidia and vfio coexist. As said nouveau and vfio can coexist. Also an interesting side note, radeon can, amdgpu can not, so far in my experience.

Generally, a milestone list
1. Computer boots to a happy intel desktop
2. Install nvidia, and it doesn't care, still booting to the intel desktop!
3. Configure vfio, the vm, boot vm into second monitor...
4. wonder how to control it....
4 a. use virt-viewer or vmm-vm window to lock or passively grap kbm (best-full screen apps)
4 b. pass separate kbm (best-if 2 kbm's)
4 c. pass eth port and x2x or x2vnc host kbm into vm (best-windowed apps)
4 d. muck up iptables and firewalls with virt network rules
4 a+c if...
4 c if wayland is NOT used anywhere...etc

So at this point you need to re-achieve step 1.
kbm= keyboard and mouse

CwF
Global Moderator
Global Moderator
Posts: 2638
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 192 times

Re: Passthrough with vfio does not work : Debian freezes on

#14 Post by CwF »

....also, if it were me I'd start with the nvidia in the furthest slot from the cpu (after the intel desktop is happy)

That should be a pci port served by the chipset (MCH, PCH, whatever) OR it is an unencumbered 4 lane port from the cpu, doubt it. In either case it should work and demonstrate the host config is correct. Then, at that success point, move the card to the preferable slot and try again. This involves NO change of the host, the vm would need the cards address corrected, or with vmm delete the device and re-attach...

I'm crossing my fingers, but typically that obvious 16x slot is not functional with the intel graphics active. Some E3 boards bifurcate to 8x intel and 8x slot. So if yours is 16/0 or 0/16 = sorry! Use the furthest slot...

marietto
Posts: 80
Joined: 2014-07-13 16:30

Re: Passthrough with vfio does not work : Debian freezes on

#15 Post by marietto »

For me it's ok if the host (OS = debian 10) uses the intel graphic chipset on one monitor and if the guest VM OS (Windows 10) uses the nvidia card passed with vfio. This because all the CG tools that I use runs better in Windows than in Linux,but Linux is better than Windows for the security. So,linux does not need to have the best graphic card that I have. Now,the problem is that only the nvidia card with the driver is working. The monitor attached with the intel graphic chipset is turned off. Should I fix this,before do something else ? Should I make work the intel graphic card with the monitor attached ? At this point I have no xorg.conf file on the debian machine.

marietto
Posts: 80
Joined: 2014-07-13 16:30

Re: Passthrough with vfio does not work : Debian freezes on

#16 Post by marietto »

I tried to generate the xorg.conf file with this command :

mario@DESKTOP-N9UN2H3:/home/mariozio# Xorg -configure

but it didn't work :

using config file : "/root/xorg.conf.new"
using system config directory "/usr/share/X11/xorg.conf.d"
modprobe: FATAL: module fbcon not found in directory /lib/modules/4.19.0-6-amd64
intel : waited 2020 ms for i915.ko driver to load
number of created screens does not match number of detected devices
configuration failed

marietto
Posts: 80
Joined: 2014-07-13 16:30

Re: Passthrough with vfio does not work : Debian freezes on

#17 Post by marietto »

maybe the solution is this :

https://unix.stackexchange.com/question ... umber-of-d

interesting what he says : This would restrict the X server to use only the Intel iGPU and ignore the Nvidia dGPU for now, and to auto-detect all the input devices, display resolutions etc.

CwF
Global Moderator
Global Moderator
Posts: 2638
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 192 times

Re: Passthrough with vfio does not work : Debian freezes on

#18 Post by CwF »

marietto wrote:use only the Intel iGPU and ignore the Nvidia dGPU for now
Not for now, forever!

Back to that milestone list, your skipping #1. Rip out the damn nvidia card and get debian working first.
You do NOT want a xorg file anywhere!

You've taken a big bite here and are having trouble chewing!

...and at the end of the day the nvidia is a poor choice to pass to windows. The windows drivers may refuse to work leaving you with an error 43 or maybe 10, requiring more tricks to get it working. Quadro hardware and drivers will work well in a windows vm. The motherboard is not a great choice either, strike two.

Get the machine working on the intel only, otherwise strike three.

Sorry I couldn't help.

marietto
Posts: 80
Joined: 2014-07-13 16:30

Re: Passthrough with vfio does not work : Debian freezes on

#19 Post by marietto »

I'm trying to remove the nvidia driver and make work the intel driver,but I'm not able to do that.

marietto
Posts: 80
Joined: 2014-07-13 16:30

Re: Passthrough with vfio does not work : Debian freezes on

#20 Post by marietto »

I did make some progress. I've installed the kernel 5.2.0 with these commands :

# as root
echo 'deb http://ftp.de.debian.org/debian buster-backports main' | tee -a /etc/apt/sources.list
apt update
apt policy linux-image-amd64
apt install linux-image-amd64=5.2+106~bpo10+1

since it gave this error :

root@DESKTOP-N9UN2H3:/home/mariozio# apt install linux-headers-5.2.0-0.bpo.2-amd64
Configurazione di linux-headers-5.2.0-0.bpo.2-amd64 (5.2.9-2~bpo10+1)...
/etc/kernel/header_postinst.d/dkms:
Error! Bad return status for module build on kernel: 5.2.0-0.bpo.2-amd64 (x86_64)
Consult /var/lib/dkms/nvidia-current/418.74/build/make.log for more information.

I gave these commands :

sudo apt purge dkms

sudo apt autoremove

sudo apt autoclean

sudo apt install dkms

now it says : can't connect to Xorg when I'm on rescue mode. And it freezes when it says : "gnome desktop manager initialized" when I'm on the normale mode. Anyway this kernel is able to make work the intel graphic chipset embedded inside the mobo. Actually the monitor attached to the nvidia graphic card is turned off. The monitor attached to the intel graphic is turned on.

Post Reply