Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

How do I seperate a device into its own IOMMU group?

If none of the specific sub-forums seem right for your thread, ask here.
Post Reply
Message
Author
marcel58095
Posts: 8
Joined: 2022-07-05 10:45

How do I seperate a device into its own IOMMU group?

#1 Post by marcel58095 »

Hey guys,
I need to seperate a device (PCIe SATA Controller) out of its IOMMU group to passtrough it into a VM in virt manager.
I have tried the following totorial but the script didn´t worked. The Tutorial is for PoP OS because I couldn´t find one for Debian.

https://forum.level1techs.com/t/vfio-in ... aft/142287

This is the IOMMU group:
IOMMU Group 15:
07:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ee]
07:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43eb]
07:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43e9]
08:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
08:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
08:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
08:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP108 [GeForce GT 1030] [10de:1d01] (rev a1)
09:00.1 Audio device [0403]: NVIDIA Corporation GP108 High Definition Audio Controller [10de:0fb8] (rev a1)
0a:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd Device [144d:a809]
29:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9215 PCIe 2.0 x1 4-port SATA 6 Gb/s Controller [1b4b:9215] (rev 11)
2a:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)


I need to seperate the device:
29:00.0 SATA controller [0106]: Marvell Technology Group Ltd. 88SE9215 PCIe 2.0 x1 4-port SATA 6 Gb/s Controller [1b4b:9215] (rev 11)


Do you have any ideas on how I could seperate the SATA Controller?

My system:
MSI B550 A PRO
Ryzen 5 5600X
16 GB RAM
1x GTX 1070 Ti (already passtrought to the VM)
1x GTX 1030

Operating System: Debian GNU/Linux 11
KDE Plasma Version: 5.20.5
KDE Frameworks Version: 5.78.0
Qt Version: 5.15.2
Kernel Version: 5.10.0-15-amd64
OS Type: 64-bit
Processors: 12 × AMD Ryzen 5 5600X 6-Core Processor
Memory: 15.6 GiB of RAM
Graphics Processor: NVIDIA GeForce GT 1030/PCIe/SSE2

CwF
Global Moderator
Global Moderator
Posts: 2636
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 192 times

Re: How do I seperate a device into its own IOMMU group?

#2 Post by CwF »

That's a mess. That should be 10-12 groups...
ACS overrides needed?
That's why I don't touch AMD boards.

Good luck

In the past storage controllers have given me grief and behavior has changed in the last years. If possible in the MB bios disable that slots ROM so that the controller is dead on boot. Next, assuming you get it isolated, depending on its reset capabilities it can be helpful to pull it's ROM image and load it by file.

CwF
Global Moderator
Global Moderator
Posts: 2636
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 192 times

Re: How do I seperate a device into its own IOMMU group?

#3 Post by CwF »

CwF wrote: 2022-07-06 16:20 That's a mess. That should be 10-12 groups...
ACS overrides needed?
Or an updated bios? Certain bridges just simply don't work.
That's why I don't touch AMD boards.

Good luck

In the past storage controllers have given me grief and behavior has changed in the last years. If possible in the MB bios disable that slots ROM so that the controller is dead on boot. Next, assuming you get it isolated, depending on its reset capabilities it can be helpful to pull it's ROM image and load it by file.

marcel58095
Posts: 8
Joined: 2022-07-05 10:45

Re: How do I seperate a device into its own IOMMU group?

#4 Post by marcel58095 »

Unfortunately there is no way to disable a PCIe Slot in my Bios and the Bios is on the newest version.

hmm, so you think there is now way?

CwF
Global Moderator
Global Moderator
Posts: 2636
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 192 times

Re: How do I seperate a device into its own IOMMU group?

#5 Post by CwF »

marcel58095 wrote: 2022-07-06 16:46 hmm, so you think there is now way?
You didn't state the use case. Generally there is little advantage passing a storage controller. You can pass entire disk with nearly the same effect.

Not unlike vfio passing usb controllers. There is no performance difference compared to a soft pass. Unless you'd like to boot off the controller without having any particular gadget predefined.

In a similar way I have a beast that passes a scsi controller to a cage and opticals where one vm can multi-boot like bare metal. Fun, not terribly useful.

just link the diisk

Code: Select all

ln -s --target-directory=/var/lib/libvirt/images /dev/disk/by-id/whatever
add to vm from default pool as 'raw'.

marcel58095
Posts: 8
Joined: 2022-07-05 10:45

Re: How do I seperate a device into its own IOMMU group?

#6 Post by marcel58095 »

The link of the disk hasn't solved my specific problem, but i'm not even sure if the storage is my problem at all.

My situation:
I want to play some games on a Windows VM. I'm using a 7.1 Headset and there is no driver or other way to get it to work on linux, so i need to use Windows.
Dual-Boot isn't an option for me, so i need to get it to work in Virt Manager.
I have a GPU passtrough and thats working just fine. But I have random stutter while playing games.
So the game just freezes for about half a second.
I have read that some people could solve that problem with passtroughing a whole SATA-Controller so i wanted to try that. Not sure if it would change something but at the moment i'm trying everything.

My guest OS is using 4 Cores with 8 Threads (passtrought), 12GB of RAM and a 512 GB Samsung SATA SSD its booting from.

CwF
Global Moderator
Global Moderator
Posts: 2636
Joined: 2018-06-20 15:16
Location: Colorado
Has thanked: 41 times
Been thanked: 192 times

Re: How do I seperate a device into its own IOMMU group?

#7 Post by CwF »

I'd say the storage isn't the issue at all.
I use a layered qcow2 for the OS and have 2 other disk passed to my Windows, along with a vfio GPU.

Different games behave differently, but a skip is common. What is the cpu utilization, does it need so many? That's the skip! Keep in mind cpu scheduling is problematic when NO OS has complete control, the host will do what it wants, and the guest is essentially being lied to. I turn off threads in the bios (Xeons) and give a minimum number of sockets and Cores, no threading, try multi-socket and single socket and compare. It's scheduling! Pinning cpu's can help, or not. It's a complicated thing that depends on specifics.

Having run the same VM on multiple host I was rather disappointed to find newer hardware hasn't helped. In fact, old Nahelem/Westmere with a chipset controlling slots and memory oversight worked better than anything since. Generally if you give more resources to the VM you expose it to more host management that may interfere with exact timing. If you can get the power you need out of 2-4 socket/cores, hold the line there = more will likely be worse. The host will move a cores load to another core for thermal management - and it skips...

marcel58095
Posts: 8
Joined: 2022-07-05 10:45

Re: How do I seperate a device into its own IOMMU group?

#8 Post by marcel58095 »

I have changed the CPU to 3 cores/ 6 threads and i haven't felt any stutter in 15 min playing.
I will propably play some hours this evening, so i can give it a better test and I will report tomorrow.
And i am mostly playing Apex Legends and it does performe best with 8 cores.
But I will propably buy a used Ryzen 9 5900X with 12 cores/ 24 threads after the release of the new Ryzen generation, so that should give me more then enough cores for my guest OS.

marcel58095
Posts: 8
Joined: 2022-07-05 10:45

Re: How do I seperate a device into its own IOMMU group?

#9 Post by marcel58095 »

CwF wrote: 2022-07-07 16:06 I'd say the storage isn't the issue at all.
I use a layered qcow2 for the OS and have 2 other disk passed to my Windows, along with a vfio GPU.

Different games behave differently, but a skip is common. What is the cpu utilization, does it need so many? That's the skip! Keep in mind cpu scheduling is problematic when NO OS has complete control, the host will do what it wants, and the guest is essentially being lied to. I turn off threads in the bios (Xeons) and give a minimum number of sockets and Cores, no threading, try multi-socket and single socket and compare. It's scheduling! Pinning cpu's can help, or not. It's a complicated thing that depends on specifics.

Having run the same VM on multiple host I was rather disappointed to find newer hardware hasn't helped. In fact, old Nahelem/Westmere with a chipset controlling slots and memory oversight worked better than anything since. Generally if you give more resources to the VM you expose it to more host management that may interfere with exact timing. If you can get the power you need out of 2-4 socket/cores, hold the line there = more will likely be worse. The host will move a cores load to another core for thermal management - and it skips...


I have played for about 4 hours yesterday and haven't felt a single stutter.
Big thank you for your help.

Post Reply