2

I have 2 Dell Servers. 1 is an R720 and the other an R730. They both have quad port daughter cards supported by Dell. Both cards are RJ45 only and have (2) 10Gbe + (2) 1Gbe ports. 1 Card is an Intel X540 and the other is a Qualcomm 57800. Both cards have the latest firmware and the latest Dell supported drivers. the 10Gbe ports are using direct ethernet with a good, Cat 7 short cable. Both systems are completely updated.

After creating SMB shares on the Linux machine, running a supported subscription version of RHEL 8.5, I get full 10Gbe speeds to and from the Windows server (whether I upload, or download it transfers at rougly 1.0-1.3GB/s). On the RHEL system (the newer, faster, and more resource heavy R730), I can not get anything over 1Gbe (it mostly hangs around 67-68MB/s). On the Linux machine it makes no difference if I am uploading from the Windows R720 or downloading.

Here is the kicker. I have swapped the daughter cards between the two machines and completely rebuilt the ethernet connections, updated drivers, and worked for several days on tweaking the alternate card in the Linux box. The best I have gotten is around 72MB/s sustained. Oddly enough, when I cut back the MTU from Jumbo frames at 9000 to standard at 1500 I would get peaky performance that might spike to 80-90MB/s, AND, if I transferred multiple files I did see one sustained transfer at 140MB/s with the smaller MTU. With the Jumbo frames it is acting like it is single threading and occasionally "stalls" for a moment. It is acting as if the RSS is not working and will only single thread through 1 CPU rather than the 16 assigned for scaling.

I am stumped. Anyone else run into issues with getting 10Gbe cards to work in Red Hat, Fedora, or CentOS? This is absolutely making me crazy! Again, two separate cards, from two different vendors, and using two different drivers are behaving exactly the same. It is almost like there is a OS throttle in Red Hat somewhere. HELP!

I have run this SAS to SAS, NVME to NVME, SAS-HD to SAS-SSD, etc. The connection seems to be throttled to the same transfer rate... ARGGGhhhhhh!

Settings for eno1:

Supported ports: [ TP ]
Supported link modes:   100baseT/Half 100baseT/Full
                        1000baseT/Full
                        10000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes:  100baseT/Half 100baseT/Full
                        1000baseT/Full
                        10000baseT/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Link partner advertised link modes:  100baseT/Full
                                     1000baseT/Full
                                     10000baseT/Full
Link partner advertised pause frame use: Symmetric Receive-only
Link partner advertised auto-negotiation: Yes
Link partner advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: on
Port: Twisted Pair
PHYAD: 17
Transceiver: internal
MDI-X: Unknown
Supports Wake-on: g
Wake-on: d
    Current message level: 0x00000000 (0)
Link detected: yes

driver: bnx2x (Previous Intel Card used IXGBE driver - IDENTICAL issues!)
version: 1.713.36-0 storm 7.13.15.0
firmware-version: FFV15.15.08 bc 7.14.16 phy 1.45

The two cards are:

BRCM 10G/GbE 2+2P 57800-t rNDC || Intel(R) 2P X540/2P I350 rNDC

Both Dell Quad Port RJ45s for the servers.

Again, keep in mind, I have swapped the cards between the machines, completely rebuilt the Linux ethernet connections, and both machines are behaving EXACTLY the same. Windows Server gets full line speed up and down, Linux gets a really slow 1Gbe (closer to 500Mbe) up and down. I do have a 500Mb Internet connection but that is running on the 1Gbe ports.

10
  • try enabling coalescing using ethtool. Generally, you'll want to figure out why you can't get faster. sudo perf top might be useful under load, so would lsirq Dec 29, 2021 at 12:10
  • I don't know the "Dell drivers"; what I do know is that the linux kernel-integrated ixgbe is not always as well-performing as the intel-website module. On the flipside, there might be some software quality issues with that (otherwise intel would have an easy time getting that into the upstream kernel) Dec 29, 2021 at 12:12
  • 2
    Ran sudo perf top - great suggestion! Shows nettle AES encryption is the #1 overhead item. Constantly in the red as well. Is there any way to disable this encryption on a particular set of IP addresses, or a subnet. Because this is direct port to port, with no switch, and will only work app to app, there is no need for encryption on this subnet. This subnet is 100% isolated for local machine to machine activity so there is no security issue requiring encryption between machines. Thoughts??? libnettle.so.6.5 || _nettle_aes_encrypt_aes
    – Bill Wood
    Dec 29, 2021 at 18:03
  • 2
    !!! Achieved full wire speed on data transfers for all but NVME to NVME! My SAS drives are pushing 175-200MB/s, SATA SSD to SAS SSD 400-500MB/s +, NVME to NVME 500MB/s+... I think the NVME speed is related to NVME changes that are needed.
    – Bill Wood
    Jan 1, 2022 at 17:07
  • 1
    Here are the fixes: > Windows File Mounts as CIFS || linuxize.com/post/how-to-mount-cifs-windows-share-on-linux || > Remove Throttling on Win Server || docs.microsoft.com/en-us/windows-server/storage/file-server/… > CIFS or SAMBA should work. Either one is fine. It appears this may be more related to the Windows throttling for SAMBA than anything. See the links above.
    – Bill Wood
    Jan 1, 2022 at 17:08

1 Answer 1

2

The #1 issue is related to Windows SMB throttling (Client OR Server versions). To conserve resources MS has added SMB throttling and limited MTU, Here is the Microsoft correction bulletin information:

  • For SMBv3 and later versions, make sure that SMB Multichannel is enabled and working.
  • On the SMB (Windows) client, enable large MTU in SMB, and disable bandwidth throttling. To do this, run the following command (on Windows using PowerShell):

Set-SmbClientConfiguration -EnableBandwidthThrottling 0 -EnableLargeMtu 1

https://docs.microsoft.com/en-us/windows-server/storage/file-server/troubleshoot/slow-file-transfer

I mounted all of the Win Shares on Linux RHEL 8 as CIFS, however, setting them up as SAMBA Shares is probably the best route (even though the CIFS is a SAMBA variation). I used these instructions just because they are simple:

https://linuxize.com/post/how-to-mount-cifs-windows-share-on-linux/

===========

RESULTS:

Achieved full wire speed for all drives EXCEPT NVME to NVME. The NVME setup tops out at a sustained 500MB/s. Normally it should be almost 2x that for full 10Gbe wire speed, but, I can live with that for a while. The SAS HD disk trxfr is 175-200MB/s (even as high as 500MB/s through the RAID card caching at times). SATA SSD to SAS SSD runs about 400-500MB/s. The NVME is the only remaining bottleneck. I'll work through that as I have time. But for now I'm satisfied with the 500MB/s sustained speeds.

2
  • I'd suggest that your connection is limited to 500MB/sec by (for instance, protocol overheads). Unless you can demonstrate a RAM -> RAM transfer at a substantially faster speed? Jan 1, 2022 at 21:02
  • Thanks Jeremy for the feedback. I would generally accept that idea except that I get 1.0-1.3GB/s on the Windows side. I realize 1.3GB/s is > 10Gbe, but, the Dell RAID cache cards do have quite a bit of impact. They can help achieve 100%+ bandwidth for a short period. There are still some settings somewhere I am missing. BUT, for now, I'm good with the 500GB/s (or, roughly 4Gbe). The rest may be some mismatches, or sync issues on settings between the 2 cards. I haven't re-checked the Windows side in a while. I will get around to that later.
    – Bill Wood
    Jan 3, 2022 at 0:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .