measuring HDR IPoIB, and anything 10GbE or higher, copy speed

Question

In dealing with large data files, i.e. greater than 10GB, I am working with IPoIB {IP over infiniband} I need something better than the traditional 1gbps wired network speed.

Background:

• on a 1gbps wired network, an SSH secure copy speed or samba file transfer between my linux servers as well as windows pc's is 112 MB/sec. note: megabyte per second. We want faster.
• have mellanox HDR inifiniband switch and adapter cards in my linux servers;
  • HDR is 100 gbps
• I am running RHEL 7.9 x86-64 and have the latest mellanox iso drivers installed
• In RHEL network manager I have the ib0 interface working with a static IP address as Datagram and my infiniband network between the few RHEL linux servers at this point is working fine.... it is working.
• In doing a secure copy as a speed test I get a solid ~260 MB/sec, even if I have 3 servers bombarding the same destination server with a 30 GB tar file.
• my thought is a 1gbps wired network is 1000 megabits per second {divided by 8 bits in a byte} is 125 MB/sec maximum and I get 112 MB/sec (89%) so I am cool with that.
  • going from 1gbps to 100gbs infiniband I hoped to see close to a corresponding 11200 MB/s secure copy speed but I only got 260 MB/sec; 260 / 11200 = 2%... which is abysmal?

Can someone let me know what kind of speeds are realistic in this scenario, given a 100 gbps HDR infiniband? Doing ethtool ib0 reports the link at 100,000, and that along with my one scp test is all I know at this point.

Is there some limit of scp and ssh manifesting itself since I have a 100 gbps now?

What is a better speed test I can do in this scenario? The goal is eventually to get the maximum file transfer speed between multiple servers on a simple LAN, using scp and eventually nfs. I am aware of nfs over rdma but I am taking baby steps.

I would also be interested in seeing any reports (answers) for anyone doing a wired network that is 10GbE or better, which would include all those slower infinibands like QDR, EDR, FDR, and so on for what transfer speeds should be.

update: i did the following test. So it seems while I have 100 gbps network speed the transfer is limited by how fast the cpu can work on the chosen cipher (and maybe hmac). So one would not simply get an expensive infiniband setup if all you are going to do is scp. I only used scp here because of its convenience in reporting transfer speed in MB/sec.

I was the only one on both servers, in my server room. Both servers were identical in hardware, and are 4 socket servers having intel 8628 cpu's, freq = 2.9ghz.

note: on a 1 gbps traditional wired network using aes-###ctr and hmac-sha2-aes### I get a solid 112 MB/sec scp speed all day long.

Answer 1

If you're looking to use SSH for this copy, the limiting factor could very well be the cipher speed. For a typical 1 Gbps network, most ciphers will perform equivalently because the limitation is the speed of the network.

However, if you're using a faster network, then the cipher matters. Assuming that your hardware is modern amd64, then [email protected] will almost certainly be the fastest. That's because modern amd64 CPUs contain instructions to accelerate both AES and GCM, and AES-128 is faster than the more secure versions.

To compare the fastest possible AEAD ciphers on my system using OpenSSL, I get 5.8 GB/s with AES-128/GCM, 4.1 GB/s with AES-256/GCM, and 2.1 GB/s with ChaCha20/Poly1305. The limitation here is literally the speed at which your CPU can encrypt the data to send it over the wire. There is no reason to think that any other program will be faster in this case unless it does encryption in a multithreaded way, since the limitation is the speed of your CPU.

You can test the speeds with your version of OpenSSL (which is used by OpenSSH) by running commands like this:

$ openssl speed -evp aes-128-gcm
$ openssl speed -evp aes-256-gcm
$ openssl speed -evp chacha20-poly1305

If you want to try to adjust the ciphers, you could do something like this:

$ scp -o [email protected],[email protected],[email protected] \
  file user@host:file

Note that in general, if you want faster transfers, you'll either need to use some technique for transferring that can do multithreaded encryption, or you'll need to transfer it unencrypted. The latter, of course, will be faster, but exposes your data and authentication credentials to everyone on the network.

Other cryptographic parameters, such as key exchange mechanisms and public key types, won't affect the speed of transfer, although they may affect the time to set up the connection. Since you're using an AEAD here, the MAC settings are also irrelevant (and an AEAD is almost certainly faster).

If you don't see improved performance after switching ciphers, it's likely that the limiting factor is something like your disk, CPU, or RAM. It's also possible that performance could be improved by using a more modern operating system, since RHEL 7 is from 2014, and the kernel, crypto libraries, and drivers probably have some performance improvements in the last 7 years.

You can test whether the problem is your disk by doing something like this (as suggested by Martin):

# dd if=/dev/sda of=/dev/null bs=1M count=10240

That will print something resembling the maximum speed at which your disk is capable of transmitting data for a 10 GB chunk.