3

On a Supermicro A2SDi-8C+-HLN4F Atom C3758 system I see that VT-d support is enabled in the BIOS but I can't find any IOMMU groups under Linux (Fedora 31):

$ ls /sys/kernel/iommu_groups
$

The kernel logs seem to indicate that VT-d is detected:

# journalctl -k | grep 'DMAR\|iommu' | sed 's/^.*kernel: //'
ACPI: DMAR 0x000000007E1629C0 000070 (v01 INTEL  BDW      00000001 INTL 00000001)
DMAR: Host address width 39
DMAR: DRHD base: 0x000000fed90000 flags: 0x1
DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap d2008c40660462 ecap f050da
DMAR: RMRR base: 0x0000007f0fa000 end: 0x0000007f10afff
DMAR-IR: IOAPIC id 2 under DRHD base  0xfed90000 IOMMU 0
DMAR-IR: HPET id 0 under DRHD base 0xfed90000
DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
DMAR-IR: Enabled IRQ remapping in x2apic mode
iommu: Default domain type: Translated

Do I need to configure something such that I also get IOMMU groups?

Background: I'm looking into this because I can't bind an ethernet device to vfio-pci for dpdk:

# /usr/share/dpdk/usertools/dpdk-devbind.py --bind=vfio-pci 07:00.0
Error: bind failed for 0000:07:00.0 - Cannot bind to driver vfio-pci
# /usr/share/dpdk/usertools/dpdk-devbind.py --status        

Network devices using kernel driver
===================================
0000:05:00.0 'Ethernet Connection X553 1GbE 15e4' if=eno1 drv=ixgbe unused=vfio-pci *Active*
0000:05:00.1 'Ethernet Connection X553 1GbE 15e4' if=eno2 drv=ixgbe unused=vfio-pci 
0000:07:00.1 'Ethernet Connection X553 1GbE 15e4' if=eno4 drv=ixgbe unused=vfio-pci 

Other Network devices
=====================
0000:07:00.0 'Ethernet Connection X553 1GbE 15e4' unused=ixgbe,vfio-pci
# dmesg | tail -n 3
[  +0.169963] ixgbe 0000:07:00.0: complete
[  +0.000561] vfio-pci: probe of 0000:07:00.0 failed with error -22
[  +0.000040] vfio-pci: probe of 0000:07:00.0 failed with error -22
6
  • Have you loaded kernel with intel_iommu=on parameter? Jun 27, 2020 at 12:21
  • @YurijGoncharuk, no. You mean that IOMMU isn't on by default under Linux? Jun 27, 2020 at 12:32
  • Indeed looks like it, the Fedora kernel config has: # CONFIG_INTEL_IOMMU_DEFAULT_ON is not set. Jun 27, 2020 at 12:36
  • Yes, IOMMU should be enabled by hand. Jun 27, 2020 at 12:36
  • @YurijGoncharuk, if you post this as an answer I'll accept it. Jun 27, 2020 at 12:37

1 Answer 1

2

You need to add intel_iommu=on to kernel load parameters. By default this parameter is disabled (I don't know distro which set this parameter to on by default).

When you've enabled IOMMU in BIOS, then ACPI table translates to kernel DMARs and kernel init this duruing the boot.

When you've set intel_iommu=on, then kernel divide system hardware resources into IOMMU groups. It's quantity and granularity depends on hardware. See VFIO - "Virtual Function I/O" for details.

1
  • Can confirm, Fedora has the kernel option CONFIG_INTEL_IOMMU_DEFAULT_ON disabled, thus IOMMU is disabled by default. Thus, after booting with intel_iommu=on the /sys/kernel/iommu_groups directory is populated and the dpdk-devbind.py --bind=vfio-pci call succeeds. Jun 27, 2020 at 15:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .