What are host bridges vs pci bridges in lspci output?

Question

I am trying to understand the pcie topology on my system and the output of lspci.

My output for lspci and the tree of the pci topology:

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne IOMMU
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne PCIe GPP Bridge
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus
00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 51)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Cezanne Data Fabric; Function 7
01:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev c0)
02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] (rev c0)
03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT]
04:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream
05:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
05:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
05:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
05:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
05:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
05:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
06:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
07:00.0 Network controller: Intel Corporation Wireless-AC 9260 (rev 29)
08:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 26)
09:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
09:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
09:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
0a:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
0b:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function (rev c9)
0c:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Renoir Radeon High Definition Audio Controller
0c:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor
0c:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne USB 3.1
0c:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne USB 3.1
0c:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller
0d:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 81)

$ tree -d --matchdirs -I '[a-z]*'  /sys/devices/pci0000\:00//sys/devices/pci0000:00/
├── 0000:00:00.0
├── 0000:00:00.2
├── 0000:00:01.0
├── 0000:00:01.1
│   ├── 0000:00:01.1:pcie010
│   └── 0000:01:00.0
│       ├── 0000:01:00.0:pcie102
│       └── 0000:02:00.0
│           ├── 0000:02:00.0:pcie202
│           ├── 0000:02:00.0:pcie210
│           ├── 0000:03:00.0
│           └── 0000:03:00.1
├── 0000:00:01.2
│   ├── 0000:00:01.2:pcie010
│   └── 0000:04:00.0
│       ├── 0000:05:01.0
│       │   ├── 0000:05:01.0:pcie202
│       │   ├── 0000:05:01.0:pcie210
│       │   └── 0000:06:00.0
│       ├── 0000:05:04.0
│       │   ├── 0000:05:04.0:pcie202
│       │   ├── 0000:05:04.0:pcie210
│       │   └── 0000:07:00.0
│       ├── 0000:05:05.0
│       │   ├── 0000:05:05.0:pcie202
│       │   ├── 0000:05:05.0:pcie210
│       │   └── 0000:08:00.0
│       ├── 0000:05:08.0
│       │   ├── 0000:05:08.0:pcie210
│       │   ├── 0000:09:00.0
│       │   ├── 0000:09:00.1
│       │   └── 0000:09:00.3
│       ├── 0000:05:09.0
│       │   ├── 0000:05:09.0:pcie210
│       │   └── 0000:0a:00.0
│       └── 0000:05:0a.0
│           ├── 0000:05:0a.0:pcie210
│           └── 0000:0b:00.0
├── 0000:00:02.0
├── 0000:00:08.0
├── 0000:00:08.1
│   ├── 0000:00:08.1:pcie010
│   ├── 0000:0c:00.0
│   ├── 0000:0c:00.1
│   ├── 0000:0c:00.2
│   ├── 0000:0c:00.3
│   ├── 0000:0c:00.4
│   └── 0000:0c:00.6
├── 0000:00:08.2
│   ├── 0000:00:08.2:pcie010
│   └── 0000:0d:00.0
├── 0000:00:14.0
├── 0000:00:14.3
│   └── PNP0800:00
├── 0000:00:18.0
├── 0000:00:18.1
├── 0000:00:18.2
├── 0000:00:18.3
├── 0000:00:18.4
├── 0000:00:18.5
├── 0000:00:18.6
└── 0000:00:18.7

As far as I know, PCI-bridges, for example 00:01.1, show up only on the primary bus which they are connected to. This bridge, which connects to buses 01,02,03, is only indicated in the output on bus 00. Then, as enumeration occurs, the first device on bus 01 will be given device number 00. The device 01:00.0 also is a PCI bridge, but I believe this is another PCI bridge (likewise with bus 02) while bus 03 is used exclusively by the VGA/Audio device which creates 2 functions (for VGA controller and Audio device). Is this accurate?

In order to understand host bridges, I think I need to understand the context of the lspci output first. My understanding is that there is a root complex (generally in the CPU itself) that translates the CPU bus to the peripheral space.

Is the root complex that is in the CPU represented by the 00:00.0 Host Bridge...Root Complex? I want to clarify this point for 2 reasons:

1. this is at odds with the before mentioned logic of assigning devices only on the primary bus which they are connected to. In this case, I realize there is no primary bus above bus 00, but I had been assuming this device would be treated separately and addressed directly by the CPU (and not need to be assigned a device number).

2. I don't understand why the root complex would share the same device class as other Host bridges in the output which are not Root Complexes (they are all Renoir PCIe Dummy Host Bridge).

And continuing on this point, I notice that none of those host bridges are connected to any other buses. Is this simply because the hardware isn't connected to anything and these bridges appear in the output despite being unused? And seeing that they are PCIe bridges, would the device class change to 'PCI bridge' if they were?

Answer 1

Answering my own question here:

Before, my confusion was due to not fundamentally understanding how PCIe works. Learning about PCIe and how a topology is enumerated cleared up most of my questions.

This article includes an image that illustrates it nicely:

The major takeaway is that every P2P connection will get its own bus number. Switches, as shown in the diagram, have 2 layers of P2P connections:

1. The upstream port (only 1 per switch) which connects whatever is upstream to its own internal bus
2. The downstream ports which connect its internal bus to whatever is downstream

The root complex is no different in the way bus numbers are enumerated. The root complex has an internal bus which will be Bus 0 every time (on a single root complex system). Anything hooked into that will be a device on Bus 0, and any P2P connections will be assigned their own bus numbers.

For an in-depth coverage of PCIe, I found the text "PCI Express Technology 3.0" published by MindShare to be valuable.

Answer 2

When it comes to bus numbering it doesn't always happen in the order you might expect.

[ken@m10plus ~]$ sudo lspci -tv
-[0000:00]-+-00.0  Intel Corporation 8th/9th Gen Core Processor Host Bridge/DRAM Registers [Coffee Lake]
           +-01.0-[07]----00.0  Sandisk Corp WD Blue SN550 NVMe SSD
           +-12.0  Intel Corporation Cannon Lake PCH Thermal Controller
           +-14.0  Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller
           +-14.2  Intel Corporation Cannon Lake PCH Shared SRAM
           +-16.0  Intel Corporation Cannon Lake PCH HECI Controller
           +-16.4  Intel Corporation Cannon Lake PCH HECI Controller #2
           +-1b.0-[06]--
           +-1c.0-[02]--+-00.0  Intel Corporation I350 Gigabit Network Connection
           |            +-00.1  Intel Corporation I350 Gigabit Network Connection
           |            +-00.2  Intel Corporation I350 Gigabit Network Connection
           |            \-00.3  Intel Corporation I350 Gigabit Network Connection
           +-1d.0-[01]--+-00.0  Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support
           |            +-00.1  Matrox Electronics Systems Ltd. MGA G200eH3
           |            +-00.2  Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging
           |            \-00.4  Hewlett-Packard Company iLO5 Virtual USB Controller
           +-1d.1-[03]--
           +-1d.2-[04]--
           +-1d.3-[05]--
           +-1f.0  Intel Corporation Device a30a
           \-1f.5  Intel Corporation Cannon Lake PCH SPI Controller
[ken@m10plus ~]$

For every system I've looked at I've found that you need to find a path to the VGA card first so that it typically bus 1. This system is an HPE gen10+ micro server, on other servers I've looked at then bus numbers seem even more mixed up.