Multiple M.2 NVMe SSDs on single slot using PCIe bifurcation: Set this up from Linux when BIOS does not?

Question

I'd like to use a multi-M.2 NVMe PCIe carrier card like an Asus Hyper M.2 x16 Card v2 in an HP Z240 tower workstation. Its C236 PCH and Skylake E3-1200V5 CPU support PCIe bifurcation of the x16 PCIe slot driven by the CPU. (Ref: Intel 100 Series / C230 Series Chipset datasheet Vol 1, p. 22; Intel Xeon E3-1200V5 datasheet Vol 1, p. 24.) Configuring the CPU PCIe for bifurcated 1x8 + 2x4 mode would allow using 3 M.2 NVMe drives in the x16 slot. (The E3 CPU is incapable of 4x4 bifurcation, so one of the M.2 slots in a quad-carrier card like the aforementioned Asus must remain empty.)

Unfortunately the Z240's BIOS Setup does not include options to configure PCIe bifurcation. Worse, it appears that HP's Sure Start dual BIOS prevents BIOS modifications which could enable PCIe bifurcation.

This Intel video introduction to PCIe bifurcation states (at 02:34) that

The configuration of the CPU PCI Express bus is statically determined by the BIOS prior to initializaton. The BIOS determines the configuration by looking at the presence detect pins on the CPU, called CFG[5] and CFG[6].

Presumably this works even if BIOS Setup doesn't include any options to configure bifurcation. But this approach would require accessing and modifying connections to the physical CPU pads, which I'd prefer to avoid.

Other BIOSes that do include options to configure bifurcation apparently override CFG[5] and CFG[6]. I have found no documentation as to how they do this, and would be grateful for any links you may be aware of.

At this point I wonder: Is there a way to override CFG[5] and CFG[6] after the machine has booted to Linux? (I realize I probably won't be able to boot from one of the M.2 drives in this case, but that's not a requirement for this system.) I'd expect such a procedure may involve steps like hot-resetting the x16, x8 and x4 PCIe controllers and/or function-level-resetting PEG Root Ports 10, 11 & 12. (Ref: Intel Xeon E3-1200V5 datasheet Vol 2.) Maybe followed by kexec?

Many thanks for any hints, tips or pointers you can provide!

Answer 1

I do not know if this is possible, but can suggest some avenues for further research:

The video you linked looks like it is for a previous generation of Intel processors (video date was 2010, and Skylake was released 2015), so it is quite likely that the Skylake processors are configured differently. That having been said, it implies that the setup is carried out by the BIOS, and that the BIOS reads the CFG pad values, and then sets up the PCIe controller configuration registers to implement that (meaning that it could ignore the CFG pad values entirely if it chooses).

Usually, the details of these types of configuration registers are only documented in the Intel "BIOS writer's guide" / "UEFI Firmware Writer's Guide" which are not publicly released by Intel for most of their CPUs. Some CPU configuration registers also include an option for the BIOS to "lock" them (so that their contents cannot be altered (or possibly even read) until the CPU is reset. If-so, the HP firmware might be doing this to prevent further changes from being made. It's also possible that the registers you need are only accessible from the CPU's "system management mode".

If I was attempting this, then I would try to find code for a SkyLake firmware (e.g. from another manufacturer) which carries out the configuration you need. You may be able to disassemble this firmware, to see how it configures the CPU to enable bifurcation, and then experiment with writing the same registers (either from Linux kernel mode, or possibly using GRUB).

You might also find some useful code in open source firmware implementations such as Coreboot.

HTH, and good luck!