10

I'm working with PCIe Gen 3 cards and from time to time they seem to fall back to PCIe 1 or 2 speeds (according to lspci and also observed by the throughput).

When rebooting/power cycling the machine the speed goes back to the full PCIe Gen 3 speed in most cases.

Is there a less intrusive way to force a renegotiation of the PCI link speed (trying to bring it back to PCI Gen 3) on e.g. RHEL6 ?

3
  • 2
    In windows I've seen it change automatically, it's a function of pci-e bus power-saving. serverfault.com/questions/226319/what-does-pcie-aspm-do seems to have a nice explanation
    – Marcin
    Commented Jul 5, 2012 at 14:07
  • thanks for this information ! Unfortunately, in my case the speed reduction happens on only some of the hosts and it does not go back to full speed when we start using the device again... Commented Jul 5, 2012 at 15:16
  • 1
    Could this be controlled by ACPI? - maybe update or install any ACPI related packages for your distribution. Alternatively maybe you can disable the feature in the BIOS.
    – LawrenceC
    Commented Aug 15, 2012 at 20:59

2 Answers 2

4

You can check your PCIe energy policy on this file:

# cat /sys/module/pcie_aspm/parameters/policy

Since Gen3 are pretty straightforward on power management through ASPM(Active-State Power Management ), this could be the root cause of the issue on your bus: The throughput is low so, the modules reduces the speed but it forgets to increase it again when needed(if needed). You could enforce on grub to avoid using the "powersave" or the "default" policy by disabling aspm with the following parameter:

pcie_aspm=off

Test this on just one kernel appending this option at /boot/grub/grub.conf on the "kernel" line of your default boot linux. Example of grub config extracted from the Red Hat docs:

default=0 
timeout=10 
splashimage=(hd0,0)/grub/splash.xpm.gz 
hiddenmenu 
title Red Hat Enterprise Linux Server (2.6.18-2.el5PAE)         
root (hd0,0)         
kernel /boot/vmlinuz-2.6.18-2.el5PAE ro root=LABEL=/1 rhgb quiet pcie_aspm=off      
initrd /boot/initrd-2.6.18-2.el5PAE.img
2
  • 2
    thanks for the answer ! In fact, in the end it turned out that the PCI card had a problem with data transmission on the PCI bus and the manufacturer fixed it with a firmware upgrade. Commented Apr 4, 2013 at 13:40
  • 2
    We had simmilar problems with a HBA at our work(brocade), and it was a firmware problem too :)
    – user34720
    Commented Apr 4, 2013 at 14:22
1

PCIe Set Speed has a pcie_set_speed.sh script which can change a device's target link speed. Have used that script successfully under Alma 8.9 (i.e. RHEL based) with a 4.18.0-513.18.1.el8_9.x86_64 Kernel.

Have a Xilinx Kintex-7 FPGA in a PCIe card:

  1. The FPGA PCIe IP was set to work at PCIe 2 speed with 4 lanes.
  2. The PCIe card is in a 16 lane slot, with an Intel i5-2310 CPU which support PCIe 2 speed.

Noticed that the PCIe interface was negotiated at PCIe 1 speed (2.5 GT/s):

$ dump_info/dump_info_libpciaccess 
domain=0000 bus=01 dev=00 func=00
  vendor_id=10ee (Xilinx Corporation) device_id=7024 ((null)) subvendor_id=0002 subdevice_id=0009
  control: I/O- Mem+ BusMaster- ParErr- SERR- DisINTx-
  status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
  bar[0] base_addr=f0110000 size=4000 is_IO=0 is_prefetchable=1 is_64=1
  bar[2] base_addr=f0100000 size=10000 is_IO=0 is_prefetchable=1 is_64=1
  Capabilities: [40] Power Management
  Capabilities: [48] Message Signaled Interrupts
  Capabilities: [60] PCI Express v2 Express Endpoint, MSI 0
    Link capabilities: Max speed 5.0 GT/s Max width x4
    Negotiated link status: Current speed 2.5 GT/s Width x4
    Link capabilities2: Not implemented
    DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
            RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
    DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
  domain=0000 bus=00 dev=01 func=00
    vendor_id=8086 (Intel Corporation) device_id=0101 (Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port) subvendor_id=8086 subdevice_id=2002
    control: I/O+ Mem+ BusMaster+ ParErr- SERR- DisINTx+
    status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
    Capabilities: [88] Bridge subsystem vendor/device ID
    Capabilities: [80] Power Management
    Capabilities: [90] Message Signaled Interrupts
    Capabilities: [a0] PCI Express v2 Root Port, MSI 0
      Link capabilities: Max speed 5.0 GT/s Max width x16
      Negotiated link status: Current speed 2.5 GT/s Width x4
      Link capabilities2: Not implemented
      DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
              RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
      DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
      SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
             Slot #0 PowerLimit 0.000W; Interlock- NoCompl+

The above output is from the dump_info_libpciaccess.c program. It starts at a given PCI device and works down the PCI topology to the root complex. The above shows both the PCIe device and root complex support 5.0 GT/s, but were running at 2.5 GT/s.

When the pcie_set_speed.sh script written by Alex Forencich was run, it reported the link speed was changed from 1 to 2. The only argument used was the PCI device to operate on, so causes the PCIe device to attempt to negotiate to it's fastest supported link speed:

$ sudo ~/Downloads/pcie_set_speed.sh 0000:01:00.0
Link capabilities: 02212d02
Max link speed: 2
Link status: 5041
Current link speed: 1
Configuring 0000:00:01.0...
Original link control 2: 00000002
Original link target speed: 2
New target link speed: 2
New link control 2: 00000002
Triggering link retraining...
Original link control: 50410040
New link control: 50410060
Link status: 5042
Current link speed: 2

And the dump_info_libpciaccess program then reported the PCIe device and root complex had negotiated at 5 GT/s:

$ dump_info/dump_info_libpciaccess 
domain=0000 bus=01 dev=00 func=00
  vendor_id=10ee (Xilinx Corporation) device_id=7024 ((null)) subvendor_id=0002 subdevice_id=0009
  control: I/O- Mem+ BusMaster+ ParErr- SERR- DisINTx-
  status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
  bar[0] base_addr=f0110000 size=4000 is_IO=0 is_prefetchable=1 is_64=1
  bar[2] base_addr=f0100000 size=10000 is_IO=0 is_prefetchable=1 is_64=1
  Capabilities: [40] Power Management
  Capabilities: [48] Message Signaled Interrupts
  Capabilities: [60] PCI Express v2 Express Endpoint, MSI 0
    Link capabilities: Max speed 5.0 GT/s Max width x4
    Negotiated link status: Current speed 5.0 GT/s Width x4
    Link capabilities2: Not implemented
    DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
            RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
    DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
  domain=0000 bus=00 dev=01 func=00
    vendor_id=8086 (Intel Corporation) device_id=0101 (Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port) subvendor_id=8086 subdevice_id=2002
    control: I/O+ Mem+ BusMaster+ ParErr- SERR- DisINTx+
    status: INTx- <ParErr- >TAbort- <TAbort- <MAbort- >SERR- DetParErr-
    Capabilities: [88] Bridge subsystem vendor/device ID
    Capabilities: [80] Power Management
    Capabilities: [90] Message Signaled Interrupts
    Capabilities: [a0] PCI Express v2 Root Port, MSI 0
      Link capabilities: Max speed 5.0 GT/s Max width x16
      Negotiated link status: Current speed 5.0 GT/s Width x4
      Link capabilities2: Not implemented
      DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
              RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
      DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
      SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
             Slot #0 PowerLimit 0.000W; Interlock- NoCompl+

I.e. a script had been successfully used to change the PCIe link speed withut needing to reboot.

Also the test_dma_bridge_parallel_streams.c program which was written to measure the DMA throughput achieved by the FPGA based PCIe device showed a increase in the throughput. The program reports the throughput achieved in the past 10 seconds. At the start of the following example the PCIe device was at a 2.5 GT/s speed, the pcie_set_speed.sh script change the PCIe device link speed while the test was running, the the measured throughput increased:

  0000:01:00.0 0 -> 1 381.186 Mbytes/sec (3811835904 bytes in 9.999928 secs)
  0000:01:00.0 1 -> 0 381.186 Mbytes/sec (3811835904 bytes in 9.999928 secs)

  0000:01:00.0 0 -> 1 381.193 Mbytes/sec (3811966976 bytes in 10.000095 secs)
  0000:01:00.0 1 -> 0 381.193 Mbytes/sec (3811966976 bytes in 10.000095 secs)

  0000:01:00.0 0 -> 1 680.225 Mbytes/sec (6802243584 bytes in 9.999985 secs)
  0000:01:00.0 1 -> 0 680.226 Mbytes/sec (6802243584 bytes in 9.999983 secs)

  0000:01:00.0 0 -> 1 737.226 Mbytes/sec (7372210176 bytes in 9.999935 secs)
  0000:01:00.0 1 -> 0 737.219 Mbytes/sec (7372210176 bytes in 10.000023 secs)

  0000:01:00.0 0 -> 1 737.226 Mbytes/sec (7372275712 bytes in 10.000028 secs)
  0000:01:00.0 1 -> 0 737.226 Mbytes/sec (7372275712 bytes in 10.000027 secs)

For the above, was using a PC which doesn't support Secure Boot. One limitation of the pcie_set_speed.sh script is that it uses setpci to change the link speed. If secure boot is enabled, the Linux Kernel lockdown then prevents setpci from performing the writes to PCIe configuration registers which is necessary to change the link speed.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .