Using 'setpci' to change MaxReadRequest value of Intel 82599ES 10Gbit Card

Question

I have an Intel 82599ES Dual 10Gbit card being used with a Linux Mint 20.2 w/kernel 5.11.0-34-generic. I am trying to use the 'setpci' command to set the value of MaxReadRequest from 512 bytes to max 2048 bytes. Here is the lspci output:

04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-2
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 34
Region 0: Memory at e0300000 (64-bit, prefetchable) [size=512K]
Region 2: I/O ports at d000 [disabled] [size=32]
Region 4: Memory at e0400000 (64-bit, prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
    Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
    Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
    Address: 0000000000000000  Data: 0000
    Masking: 00000000  Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
    Vector table: BAR=4 offset=00000000
    PBA: BAR=4 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
    DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
    DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
        MaxPayload 256 bytes, **MaxReadReq 512 bytes**
    DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
    LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <1us
        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
    LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
    LnkSta: Speed 5GT/s (ok), Width x8 (ok)
        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
    DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, NROPrPrP-, LTR-
         10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix-
         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
         FRS-, TPHComp-, ExtTPHComp-
         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
    DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
         AtomicOpsCtl: ReqEn-
    LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

Using the Intel Datasheet for 82599, it says that the Device Control Register that controls the MaxReadReq is at Byte Offset 0xA8, specifically:

And here is the hexadecimal space of the 82599ES:

sudo lspci -xxx -s 04:00.1
04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
00: 86 80 fb 10 06 04 10 00 01 00 00 02 10 00 80 00
10: 0c 00 30 e0 00 00 00 00 01 d0 00 00 00 00 00 00
20: 0c 00 40 e0 00 00 00 00 00 00 00 00 86 80 03 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 02 00 00
40: 01 50 23 48 00 20 00 2b 00 00 00 00 00 00 00 00
50: 05 70 80 01 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 11 a0 3f 80 04 00 00 00 04 20 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 10 00 02 00 c2 8c 00 10 3f 28 09 00 82 c4 01 00
b0: 40 00 82 10 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 1f 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

I've found an example from Mellanox' Understanding PCIe Configuration For Max Performance but using the setpci command that they are using doesn't seem to do anything...probably since I am changing the wrong register location. I am still struggling on how to make sense of "byte offset 0xA8" at bits 14:12 and to transcribe that into something that setpci would understand...

Question: Using the 'setpci' command, how do you figure out what is 0xA8, bit 14:12, and change its value to "100b" to change the MaxReadReq from 512 bytes to 2048 bytes ? Any help much appreciated!

Answer 1

setpci -s 04:00.1 a8.w will output the value of the Device Control Register. 

• -s 04:00.1 is the device specification, as shown in the question.
• a8 is the address of the Device Control Register, as shown in the question.
• .w specifies that you are accessing a two-byte (word) register.

See the setpci(8) man page for more information.

As you know, bits 12-14 will represent the Max Read Request Size in a logarithmic mapping.  As shown in the Intel Datasheet for 82599 (quoted in the question),

• 0xxx →  128
• 1xxx →  256
• 2xxx →  512
• 3xxx → 1024
• 4xxx → 2048

With whatever value the adapter returned for the last 3 digits, change the first digit and set the new value using the name=value syntax.  E.g., if the adapter returned 283f, and you wanted to set it to 2048 bytes, then you will change the value to 483f.

setpci -s 04:00.1 a8.w=483f

Check that it took your value by re-issuing without a value to try the read again:

setpci -s 04:00.1 a8.w

It should return the value you entered.

Power off and reboot to make active.