"AMD IOMMUv2 functionality not available on this system" When it should be

[abbatrombone]

I have been setting up a new VM on my machine and been running into an issue.

Code: Select all

$ sudo dmesg | grep IOMMU
[    0.022494] DMAR: IOMMU enabled
[    2.191199] AMD-Vi: AMD IOMMUv2 functionality not available on this system - This is not a bug.

I have edited /etc/default/grub to have GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=1 amd_iommu=on" then run sudo update-grub, and then reboot.

I have dine the other steps past this like:
I've been able to verify the GPU (VGA and audio) ids as 1002:ab28,1002:743f

I know my cpu (intel 12700kf) should support this

Code: Select all

$ ps -ef | grep amd_iommu_v2
usr+    9164    8614  0 18:46 pts/0    00:00:00 grep amd_iommu_v2

update /etc/initramfs-tools/modules by adding

fio
vfio_iommu_type1
vfio_virqfd
options vfio_pci ids=1002:ab28,1002:743f
vfio_pci ids=1002:ab28,1002:743f
vfio_pci
amdgpu

update /etc/modules by adding the following

vfio
vfio_iommu_type1
vfio_pci ids=1002:ab28,1002:743f

creating and adding /etc/modprobe.d/amdgpu.conf
softdep amdgpu pre: vfio vfio_pci

as well as /etc/modprobe.d/vfio_pci.conf
options vfio_pci ids=1002:67df,1002:aaf0

This still leads to second GPUs second driver to be amdgpu rather than vfio-pci. Im pretty sure this is due to "AMD IOMMUv2 functionality not available on this system - This is not a bug."

Code: Select all

$ lspci -vnn | grep -iP "vga|amdgpu|nvidia|nouveau|vfio-pci"
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6950 XT] [1002:73a5] (rev c0) (prog-if 00 [VGA controller])
	Kernel driver in use: amdgpu
	Kernel modules: amdgpu
09:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 24 [Radeon RX 6400/6500 XT/6500M] [1002:743f] (rev c1) (prog-if 00 [VGA controller])
	Kernel driver in use: amdgpu
	Kernel modules: amdgpu

[CwF]

"AMD IOMMUv2 functionality not available on this system - This is not a bug."

It has nothing to do with your system.

/etc/modprobe.d/vfio_pci.conf is the only place you need to declare the device id's, and yours are wrong.

[1002:73a5]

You shouldn't need the pre: in /etc/modprobe.d/amdgpu.conf at all.

[abbatrombone]

I had added to my /etc/modprobe.d/vfio_pci.conf

Code: Select all

$ sudo nano /etc/modprobe.d/vfio_pci.conf 

#This was added Dec 2nd for GPU passthrough

options vfio_pci ids=1002:ab28,1002:743f

I thought the IDs were correct. I want the Radeon RX 6950 XT for the host and the Radeon RX 6400/6500 XT/6500M for the VM. So shouldnt i be using

1002:743f

?

Here are the IDs i find

Code: Select all

lspci -nn | grep -iP "VGA|audio"
00:1f.3 Audio device [0403]: Intel Corporation Alder Lake-S HD Audio Controller [8086:7ad0] (rev 11)
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6950 XT] [1002:73a5] (rev c0)
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]
09:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 24 [Radeon RX 6400/6500 XT/6500M] [1002:743f] (rev c1)
09:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]

[CwF]

I thought the IDs were correct.

You're right, I was mistaken.

I assume virtualization is enabled in the bios?

[abbatrombone]

I have been able to get the VM to run without GPU pass through so it is enabled. I went in a checked and It is enabled. I forgot to write down what it was called in the bios. I do have the card in the bottom slot in my machine, there is another x16 slot above it, but i dont think thats the issue.

[CwF]

A cleaner listing;

Code: Select all

lspci -bnkd 1002:

check was is getting registered;

Code: Select all

journalctl --no-pager | grep vfio

and I apologize but caught it, that's old school there, now a better:

Code: Select all

journalctl -b 0 -g vfio --no-pager

Do check slots, and also their settings in the bios. A quick answer to 'are all slots the same' is an easy NO

The block diagram to the motherboard would give a clue.

[abbatrombone]

Here's the code output for what you send:

Code: Select all

trazodone@debian:~$ lspci -bnkd 1002:
01:00.0 0604: 1002:1478 (rev c0)
	Kernel driver in use: pcieport
02:00.0 0604: 1002:1479
	Kernel driver in use: pcieport
03:00.0 0300: 1002:73a5 (rev c0)
	Subsystem: 1849:5238
	Kernel driver in use: amdgpu
	Kernel modules: amdgpu
03:00.1 0403: 1002:ab28
	Subsystem: 1002:ab28
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
07:00.0 0604: 1002:1478 (rev c1)
	Kernel driver in use: pcieport
08:00.0 0604: 1002:1479
	Kernel driver in use: pcieport
09:00.0 0300: 1002:743f (rev c1)
	Subsystem: 1462:5080
	Kernel driver in use: amdgpu
	Kernel modules: amdgpu
09:00.1 0403: 1002:ab28
	Subsystem: 1002:ab28
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
trazodone@debian:~$ journalctl --no-pager | grep vfio
Hint: You are currently not seeing messages from other users and the system.
      Users in groups 'adm', 'systemd-journal' can see all messages.
      Pass -q to turn off this notice.
^C
trazodone@debian:~$ journalctl -b 0 -g vfio --no-pager
Hint: You are currently not seeing messages from other users and the system.
      Users in groups 'adm', 'systemd-journal' can see all messages.
      Pass -q to turn off this notice.
-- No entries --

I put the card in the bottom one MoBo Manuel it says its a PCIEX4_2 so its only 4 lanes then. It having fewer lanes would reduce preformance but it should work right?

[CwF]

@abbatrombone
Yes, even a single lane can work. But this is a single factor, some cards yes, some no. Often the sub components on the way to the slot also matter; the pci root port, bifurcation switches, bus conversion bridges etc. Any one ingredient can bork the process.

But, none of that matters yet,
Do run the journalctl check to see if your declarations are being registered. If not, we need to backtrack to those and fix how you are doing that.
I only have old examples, the journalctl way is current;

Code: Select all

:~# dmesg | grep -i vfio
[    6.354541] VFIO - User Level meta-driver version: 0.3
[    6.394363] vfio_pci: add [1002:683f[ffff:ffff]] class 0x000000/00000000
[    6.410211] vfio_pci: add [1002:aab0[ffff:ffff]] class 0x000000/00000000
[    6.410225] vfio_pci: add [1b73:1100[ffff:ffff]] class 0x000000/00000000
[    6.410237] vfio_pci: add [1000:0030[ffff:ffff]] class 0x000000/00000000
[    6.410249] vfio_pci: add [8086:105e[ffff:ffff]] class 0x000000/00000000
[    6.410257] vfio_pci: add [8086:3a3c[ffff:ffff]] class 0x000000/00000000
[    6.410265] vfio_pci: add [8086:3a37[ffff:ffff]] class 0x000000/00000000
[    6.410273] vfio_pci: add [8086:3a38[ffff:ffff]] class 0x000000/00000000
[    6.410281] vfio_pci: add [8086:3a39[ffff:ffff]] class 0x000000/00000000

Until you have a listing like this, focus on your declarations.

remove “fio” and pci_id lines in /etc/initramfs-tools/modules
remove /etc/modprobe.d/amdgpu.conf
softdep amdgpu pre: vfio vfio_pci should be in /etc/initramfs-tools/modules if needed.
in /etc/modprobe.d/vfio_pci.conf change vfio_pci, it's not an underscore

Code: Select all

options vfio-pci ids=1002:ab28,1002:743f

then update-initramfs

[abbatrombone]

So i ran the command it seems some are being registared but some have errors

Code: Select all

sudo dmesg | grep -i vfio
[sudo] password for trazodone: 
[    2.473263] VFIO - User Level meta-driver version: 0.3
[    2.478903] vfio-pci: probe of 0000:03:00.1 failed with error -22
[    2.478987] vfio-pci: probe of 0000:09:00.1 failed with error -22
[    2.478988] vfio_pci: add [1002:ab28[ffffffff:ffffffff]] class 0x000000/00000000
[    2.478993] vfio-pci: probe of 0000:03:00.1 failed with error -22
[    2.478999] vfio-pci 0000:09:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[    2.479011] vfio-pci: probe of 0000:09:00.0 failed with error -22
[    2.479015] vfio-pci: probe of 0000:09:00.1 failed with error -22
[    2.479016] vfio_pci: add [1002:743f[ffffffff:ffffffff]] class 0x000000/00000000
[   28.602759]  snd_usb_audio sha512_generic snd_hda_codec cryptd iwlwifi videobuf2_vmalloc ctr snd_usbmidi_lib videobuf2_memops snd_hda_core rapl videobuf2_v4l2 snd_rawmidi videobuf2_common drbg iTCO_wdt snd_hwdep snd_seq_device intel_cstate cfg80211 ansi_cprng snd_pcm intel_pmc_bxt ecdh_generic intel_uncore videodev iTCO_vendor_support mei_me rfkill snd_timer gigabyte_wmi wmi_bmof watchdog snd mei pcspkr mc ecc soundcore intel_hid joydev intel_pmc_core sparse_keymap acpi_tad acpi_pad sg evdev nfsd msr auth_rpcgss parport_pc nfs_acl ppdev lockd lp grace parport sunrpc fuse loop dm_mod efi_pstore configfs efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic hid_generic usbhid hid sd_mod amdgpu vfio_pci vfio_pci_core vfio_virqfd irqbypass vfio_iommu_type1 vfio gpu_sched drm_buddy i2c_algo_bit drm_display_helper cec rc_core drm_ttm_helper ttm drm_kms_helper nvme ahci nvme_core libahci xhci_pci libata xhci_hcd t10_pi drm mxm_wmi usbcore scsi_mod igc crc64_rocksoft
[37618.867133]  snd_usb_audio sha512_generic snd_hda_codec cryptd iwlwifi videobuf2_vmalloc ctr snd_usbmidi_lib videobuf2_memops snd_hda_core rapl videobuf2_v4l2 snd_rawmidi videobuf2_common drbg iTCO_wdt snd_hwdep snd_seq_device intel_cstate cfg80211 ansi_cprng snd_pcm intel_pmc_bxt ecdh_generic intel_uncore videodev iTCO_vendor_support mei_me rfkill snd_timer gigabyte_wmi wmi_bmof watchdog snd mei pcspkr mc ecc soundcore intel_hid joydev intel_pmc_core sparse_keymap acpi_tad acpi_pad sg evdev nfsd msr auth_rpcgss parport_pc nfs_acl ppdev lockd lp grace parport sunrpc fuse loop dm_mod efi_pstore configfs efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic hid_generic usbhid hid sd_mod amdgpu vfio_pci vfio_pci_core vfio_virqfd irqbypass vfio_iommu_type1 vfio gpu_sched drm_buddy i2c_algo_bit drm_display_helper cec rc_core drm_ttm_helper ttm drm_kms_helper nvme ahci nvme_core libahci xhci_pci libata xhci_hcd t10_pi drm mxm_wmi usbcore scsi_mod igc crc64_rocksoft
[37639.711369]  snd_usb_audio sha512_generic snd_hda_codec cryptd iwlwifi videobuf2_vmalloc ctr snd_usbmidi_lib videobuf2_memops snd_hda_core rapl videobuf2_v4l2 snd_rawmidi videobuf2_common drbg iTCO_wdt snd_hwdep snd_seq_device intel_cstate cfg80211 ansi_cprng snd_pcm intel_pmc_bxt ecdh_generic intel_uncore videodev iTCO_vendor_support mei_me rfkill snd_timer gigabyte_wmi wmi_bmof watchdog snd mei pcspkr mc ecc soundcore intel_hid joydev intel_pmc_core sparse_keymap acpi_tad acpi_pad sg evdev nfsd msr auth_rpcgss parport_pc nfs_acl ppdev lockd lp grace parport sunrpc fuse loop dm_mod efi_pstore configfs efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic hid_generic usbhid hid sd_mod amdgpu vfio_pci vfio_pci_core vfio_virqfd irqbypass vfio_iommu_type1 vfio gpu_sched drm_buddy i2c_algo_bit drm_display_helper cec rc_core drm_ttm_helper ttm drm_kms_helper nvme ahci nvme_core libahci xhci_pci libata xhci_hcd t10_pi drm mxm_wmi usbcore scsi_mod igc crc64_rocksoft
[37649.877808]  snd_usb_audio sha512_generic snd_hda_codec cryptd iwlwifi videobuf2_vmalloc ctr snd_usbmidi_lib videobuf2_memops snd_hda_core rapl videobuf2_v4l2 snd_rawmidi videobuf2_common drbg iTCO_wdt snd_hwdep snd_seq_device intel_cstate cfg80211 ansi_cprng snd_pcm intel_pmc_bxt ecdh_generic intel_uncore videodev iTCO_vendor_support mei_me rfkill snd_timer gigabyte_wmi wmi_bmof watchdog snd mei pcspkr mc ecc soundcore intel_hid joydev intel_pmc_core sparse_keymap acpi_tad acpi_pad sg evdev nfsd msr auth_rpcgss parport_pc nfs_acl ppdev lockd lp grace parport sunrpc fuse loop dm_mod efi_pstore configfs efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic hid_generic usbhid hid sd_mod amdgpu vfio_pci vfio_pci_core vfio_virqfd irqbypass vfio_iommu_type1 vfio gpu_sched drm_buddy i2c_algo_bit drm_display_helper cec rc_core drm_ttm_helper ttm drm_kms_helper nvme ahci nvme_core libahci xhci_pci libata xhci_hcd t10_pi drm mxm_wmi usbcore scsi_mod igc crc64_rocksoft
[37665.195480]  snd_usb_audio sha512_generic snd_hda_codec cryptd iwlwifi videobuf2_vmalloc ctr snd_usbmidi_lib videobuf2_memops snd_hda_core rapl videobuf2_v4l2 snd_rawmidi videobuf2_common drbg iTCO_wdt snd_hwdep snd_seq_device intel_cstate cfg80211 ansi_cprng snd_pcm intel_pmc_bxt ecdh_generic intel_uncore videodev iTCO_vendor_support mei_me rfkill snd_timer gigabyte_wmi wmi_bmof watchdog snd mei pcspkr mc ecc soundcore intel_hid joydev intel_pmc_core sparse_keymap acpi_tad acpi_pad sg evdev nfsd msr auth_rpcgss parport_pc nfs_acl ppdev lockd lp grace parport sunrpc fuse loop dm_mod efi_pstore configfs efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic hid_generic usbhid hid sd_mod amdgpu vfio_pci vfio_pci_core vfio_virqfd irqbypass vfio_iommu_type1 vfio gpu_sched drm_buddy i2c_algo_bit drm_display_helper cec rc_core drm_ttm_helper ttm drm_kms_helper nvme ahci nvme_core libahci xhci_pci libata xhci_hcd t10_pi drm mxm_wmi usbcore scsi_mod igc crc64_rocksoft
[47369.045551]  snd_usb_audio sha512_generic snd_hda_codec cryptd iwlwifi videobuf2_vmalloc ctr snd_usbmidi_lib videobuf2_memops snd_hda_core rapl videobuf2_v4l2 snd_rawmidi videobuf2_common drbg iTCO_wdt snd_hwdep snd_seq_device intel_cstate cfg80211 ansi_cprng snd_pcm intel_pmc_bxt ecdh_generic intel_uncore videodev iTCO_vendor_support mei_me rfkill snd_timer gigabyte_wmi wmi_bmof watchdog snd mei pcspkr mc ecc soundcore intel_hid joydev intel_pmc_core sparse_keymap acpi_tad acpi_pad sg evdev nfsd msr auth_rpcgss parport_pc nfs_acl ppdev lockd lp grace parport sunrpc fuse loop dm_mod efi_pstore configfs efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic hid_generic usbhid hid sd_mod amdgpu vfio_pci vfio_pci_core vfio_virqfd irqbypass vfio_iommu_type1 vfio gpu_sched drm_buddy i2c_algo_bit drm_display_helper cec rc_core drm_ttm_helper ttm drm_kms_helper nvme ahci nvme_core libahci xhci_pci libata xhci_hcd t10_pi drm mxm_wmi usbcore scsi_mod igc crc64_rocksoft

For /etc/initramfs-tools/modules do you want this?

vfio
vfio_iommu_type1
vfio_virqfd
vfio_pci
amdgpu

made the change to options vfio-pci ids=1002:ab28,1002:743f
options vfio-pci ids=1002:ab28,1002:743f

[CwF]

For /etc/initramfs-tools/modules do you want this?

vfio
vfio_iommu_type1
vfio_virqfd
vfio_pci
amdgpu

Those are fine, amdgpu isn't needed. This changes according to the kernel defaults and I haven't checked lately.

[ 2.478903] vfio-pci: probe of 0000:03:00.1 failed with error -22
[ 2.478987] vfio-pci: probe of 0000:09:00.1 failed with error -22

This is a problem. The declarations are being added, and it doesn't look like your slots are compatible. The only time I've seen the error is with a bridge chip in the way:

Code: Select all

[    7.876944] vfio-pci: probe of 0000:06:00.0 failed with error -22
===	06:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8114 PCI Express-to-PCI/PCI-X Bridge [10b5:8114] (rev bc)

Since both sound devices are the same, you might try removing the audio part and see if it makes a difference. We should pass the whole IOMMU group, but ?

Ok, two more things,
sudo dmesg | grep -i vfio == sudo journalctl -b 0 -g vfio --no-pager

With this command, look for you card and make sure it is listed with [RESET], and in a clean IOMMU group

Code: Select all

# for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done

and a response like;

Code: Select all

IOMMU group 15
	03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750/8740 / R7 250E] [1002:683f]
[RESET]	03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]

So, not sure I have any more to offer, sorry.

[abbatrombone]

So I ran

Code: Select all

for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done

andddddddddddddddddddddddd 0 output.

I think im just gonna call this a failure. thanks for all your support. I at the very least learned a lot even if it ended up failing. Its an advanced subject afterall. I should remove the changes i assume. update-grub and update-initramfs, shut down, and remove the extra GPU.

[CwF]

update-grub

hold up, maybe not
this reminded me to glance at your earlier post, your grub line is also wrong, and I skipped that little detail. Fix that and you should be good.

Code: Select all

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

[abbatrombone]

I tried that not luck

Code: Select all

lspci -vnn | grep -iP "vga|amdgpu|nvidia|nouveau|vfio-pci"
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6950 XT] [1002:73a5] (rev c0) (prog-if 00 [VGA controller])
	Kernel driver in use: amdgpu
	Kernel modules: amdgpu
09:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 24 [Radeon RX 6400/6500 XT/6500M] [1002:743f] (rev c1) (prog-if 00 [VGA controller])
	Kernel driver in use: amdgpu
	Kernel modules: amdgpu

[arzgi]

What did you try? You have amd, not intel.

[abbatrombone]

If i tried it, its in this forum chain. I do Have an intel CPU, but an AMD GPU.