1

I bought AMD RyZEN 5 2400G, Gigabyte B450M D3SH and 16GB RAM. Some elder GPU AMD Radeon HD 6XXX I took for passing through. But I was unpleasantly surprised with the IOMMU grouping.

I put the GPU in either first (16/16) or third (16/8) PCIe slot, and found that it was in both cases included in the iommu group zero. But other members of this iommu group cannot be passed though to VM neither excluded from host system. There were just four iommu groups (0 - 3), while in the tutorials and my other PCs I see twelve or more groups, and some PCIe slots are quite isolated in a single group. Questions: Does anybody knows:

  • If I should buy some other hardware?
  • If there are some kernel options to get more iommu groups (split)?
  • If do I have to use legacy BIOS instead of uEFI?

SVM enabled, IOMMU enabled. OS: Debian Linux testitng, kernel 5.10, kernel cmd options iommu=on amd_iommu=on

iommu group zero:

01:00.0 USB       [0c03]: [AMD] 400 Series Chipset USB 3.1 XHCI Controller [1022:43d5] (rev 01)
01:00.1 SATA      [0106]: [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
01:00.2 PCIbridge [0604]: [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01)
02:00.0 PCIbridge [0604]: [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
04:00.0 Ethernet  [0200]: Realtek Co. Ltd. RTL8111/8168/8411 PCIe Gigabit Eth [10ec:8168] (rev 0c)
05:00.0 VGA       [0300]: [AMD/ATI] Turks PRO [Radeon HD 6570/7570/8550] [1002:6759]
05:00.1 Audio dev [0403]: [AMD/ATI] Turks HDMI Audio [Radeon HD 6500/6600/6700M Series] [1002:aa90]
06:00.0 NVMe      [0108]: Intel NVMe Optane Memory [8086:2522]

group 1

...
08:00.0 VGA       [0300]: [AMD/ATI] Raven Ridge [Radeon Vega (Mobile) Series] [1002:15dd] (rev c6)
08:00.1 Audio dev [0403]: [AMD/ATI] Raven/Raven2/Fenghuang HDMI/DP Audio Controller [1002:15de]
...

As you can see, the APU (GPU integrated on the CPU chip) is a member of the group 1.

2 Answers 2

2

Well, it seems to be a very complicated question. It was a really hard nut. First I suspected the OS, because I installed Debian testing (Bullseye), what is somewhat more dangerous than stable version. Hence I googled for some kernel patch. Yes, the patch exists, but it is not recommended because of security break. Then I googled for the detail specification of chipset and motherboard IOMMU options. Such information are apparently not important for the customers neither the potential buyers.

After some hopeless days, I restarted with the uEFI BIOS setup, step by step. And finally I got it: The settings touching the virtualisation are spread into various (sub)menu items. So you have to check:

MIT\Advanced Freq. Settings\Advanced CPU Settings\SVM Mode = Enabled !
Chipset\IOMMU = Enabled !
Peripherals\AMD CBS\PCIe ARI Support = Disable !

The last -PCIe ARI Support- is for the IOMMU Group separation very important. I did not found a detail description of this BIOS option. But after correct setup, now I have 12 IOMMU groups and the GPU put in PCIe slot 1 is in a single group 9 ( GPU and associated Audio).

I hope, this post, even I wrote the answer myself, can help somebody a little.

Edit 2021-03-16: In recent time I got an explanation (for lama) of the ARI: old addressing of PCI uses an address and subaddress, let's say the address count 5 bits and subaddress 3 bits. Hence you can use just 32 diferent PCI devices but each can hold 8 subdevices (e.g. GPU and Audio). ARI agregates the 5 and 3 address bits into 8 bits address without subaddresses, hence you can have 256 diferent PCI devices. But it dramatially changes the grouping. Some sources says that in near past the default config of uEFI BIOS has been changed to ARI disabled.

0

As the first answer, it's very complicated!

Simple answer is: A PCIe device with ACS (Access Control Services) capability, then has owner IOMMU group; without ACS, it is dependent on the upstream device.

The ACS capability depends on its provider, such RyZEN CPU with and chipset without.

But some flavours of kernel change that, you can try to search "pve unraid iommu group patch".

Finally, thanks your answer about "PCIe ARI Support", it helps me. But you do not need to change the BIOS, only append kernel param as pci=noari to do that.

2
  • While configuring a new Gigabyte/RyZen MoBo, the trick pci=noari did not gave expected result with Debian Bullseye kernel. Not really sure, if the kernel can securely change the behavior of setup configured PCI. NB: Even the same MoBo vendor and uEFI BIOS producer, the virtualisation config is spread again into unexpectable submenues.
    – schweik
    Sep 13, 2022 at 10:32
  • Just an important note. The succesless looking for correct combination of BIOS parameters lead to basic check the compatibility list of BIOS version with CPU version. After BIOS version update the IOMMU groups was immediately correct.
    – schweik
    Sep 16, 2022 at 18:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .