3

I'd like to use a multi-M.2 NVMe PCIe carrier card like an Asus Hyper M.2 x16 Card v2 in an HP Z240 tower workstation. Its C236 PCH and Skylake E3-1200V5 CPU support PCIe bifurcation of the x16 PCIe slot driven by the CPU. (Ref: Intel 100 Series / C230 Series Chipset datasheet Vol 1, p. 22; Intel Xeon E3-1200V5 datasheet Vol 1, p. 24.) Configuring the CPU PCIe for bifurcated 1x8 + 2x4 mode would allow using 3 M.2 NVMe drives in the x16 slot. (The E3 CPU is incapable of 4x4 bifurcation, so one of the M.2 slots in a quad-carrier card like the aforementioned Asus must remain empty.)

Unfortunately the Z240's BIOS Setup does not include options to configure PCIe bifurcation. Worse, it appears that HP's Sure Start dual BIOS prevents BIOS modifications which could enable PCIe bifurcation.

This Intel video introduction to PCIe bifurcation states (at 02:34) that

The configuration of the CPU PCI Express bus is statically determined by the BIOS prior to initializaton. The BIOS determines the configuration by looking at the presence detect pins on the CPU, called CFG[5] and CFG[6].

Presumably this works even if BIOS Setup doesn't include any options to configure bifurcation. But this approach would require accessing and modifying connections to the physical CPU pads, which I'd prefer to avoid.

Other BIOSes that do include options to configure bifurcation apparently override CFG[5] and CFG[6]. I have found no documentation as to how they do this, and would be grateful for any links you may be aware of.

At this point I wonder: Is there a way to override CFG[5] and CFG[6] after the machine has booted to Linux? (I realize I probably won't be able to boot from one of the M.2 drives in this case, but that's not a requirement for this system.) I'd expect such a procedure may involve steps like hot-resetting the x16, x8 and x4 PCIe controllers and/or function-level-resetting PEG Root Ports 10, 11 & 12. (Ref: Intel Xeon E3-1200V5 datasheet Vol 2.) Maybe followed by kexec?

Many thanks for any hints, tips or pointers you can provide!

  • I asked a similar question on reddit. I'd like to use more network cards on a split x16 slot. It appears to be that the c236 chipset and the CPU support is not enough, the BIOS of the motherboard must support it too. I have an MSI c236m motherboard, I just asked MSI support if their BIOS contains it. Even if so, I am not sure if I can find a proper PCIE splitter card anywhere, since this is not a common problem and the best would be 1x16->4x4 for me instead of 1x16->2x8... Did you have any success? – inf3rno Jun 13 at 2:56
  • @inf3rno: I've no progress to report, unfortunately. Could you please add a link to your reddit thread? You have an interesting application in mind. I assume you're aware of quad 1Gbe cards like Intel I350-T4? Your MSI motherboard is more likely to accept a BIOS mod than the locked-down one in my HP. Do be aware that 4x4 bifurcation is not supported by the hardware; there are only 3 PCIe controllers on the x16 slot driven by the CPU. – fmyhr Jun 14 at 13:33
  • I read a lot since I last wrote. Looks like using my server for switching and routing does not have a massive price advantage. There is a cheap Mikrotik SFP+ switch everybody recommends: amazon.com/MikroTik-CRS305-1G-4S-Gigabit-Ethernet-RouterOS/dp/… So I rather buy a switch for a little extra money if I need to connect multiple computers. – inf3rno Jun 15 at 8:01
  • It turned out my CPU supports only 16 lanes and 8 are taken by the video card. So I could add a 4 port x8 PCIE card at most. It would cost the same as the upper 4 port switch. All I would spare is the price of an 1 port PCIE card, 2 transceivers and 1 cable. Maybe $100 total with cheap second hand parts. So I would spare some money with it, but it would take a lot of time and effort to turn my server into a switch. – inf3rno Jun 15 at 8:15
2

I do not know if this is possible, but can suggest some avenues for further research:

The video you linked looks like it is for a previous generation of Intel processors (video date was 2010, and Skylake was released 2015), so it is quite likely that the Skylake processors are configured differently. That having been said, it implies that the setup is carried out by the BIOS, and that the BIOS reads the CFG pad values, and then sets up the PCIe controller configuration registers to implement that (meaning that it could ignore the CFG pad values entirely if it chooses).

Usually, the details of these types of configuration registers are only documented in the Intel "BIOS writer's guide" / "UEFI Firmware Writer's Guide" which are not publicly released by Intel for most of their CPUs. Some CPU configuration registers also include an option for the BIOS to "lock" them (so that their contents cannot be altered (or possibly even read) until the CPU is reset. If-so, the HP firmware might be doing this to prevent further changes from being made. It's also possible that the registers you need are only accessible from the CPU's "system management mode".

If I was attempting this, then I would try to find code for a SkyLake firmware (e.g. from another manufacturer) which carries out the configuration you need. You may be able to disassemble this firmware, to see how it configures the CPU to enable bifurcation, and then experiment with writing the same registers (either from Linux kernel mode, or possibly using GRUB).

You might also find some useful code in open source firmware implementations such as Coreboot.

HTH, and good luck!

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.