scp performance over 1Gbit LAN

Question

On my wired LAN, with 1GBit/s devices, I have two Linux machines (One Haswell, One Skylake Xeon) and when I do a secure copy of a large file, I see 38MB/s.

Seeing that this is 3 times below the 1000Mbit/s spec, I wonder if this performance is as expected?

Both machines use SSD for storage, both run 64bit Ubuntu.

During the transfer, both machine have approximately one core at 30% load.

The router that sits between the machines is a TP-Link Archer C7 AC1750. Both machines have Intel(R) Gigabit Ethernet Network devices that are in Full Duplex mode.

What is a normal scp transfer speed on 1Gbit LANs?

UPDATE

• Using /dev/zero to rule out disk IO yielded the same results.
• Using nc yielded slightly higher: 41MiB/s.
• Paradoxically, UDP nc was slower than TCP nc, at 38MiB/s?
• Switching to crossover cable: 112MB/s for scp.

CONCLUSION

The TP-Link router in between was the weak link in the network, and could not keep up.

Answer 1

It does seem slow from a theoretical stand point although I've not seen any transfers much quicker practically on home hardware.

Some experiments you might like to try to rule out possible limiting factors:

• Assess your raw SSH speed by copying from /dev/zero to /dev/null. This rules out a HD bottle neck.

  ssh remote_host cat /dev/zero | pv > /dev/null
  

• Check other unencrypted protocols such as HTTP. HTTP actually sends files as with nothing but a header. Sending large files over HTTP is a reasonable measure of TCP speeds.
• Check that you are not forcing traffic through the router but only it's ethernet switch. For example if your machine has a public IP and local IP, then scp to/from the local IP. This is because home routers often have to process WAN traffic through their CPU which creates a bottle neck. Even if both machines are on the LAN, using the public IP can force packets through the CPU as if it was going to the WAN.
• Similarly I would use IPv4. Some home routers have a weird behaviour with IPv6 where they ask all local traffic to be forwarded to the router.
• If at all possible try with a gigabit crossover cable instead of a router. This should rule out router.

Answer 2

What is a normal scp transfer speed on 1Gbit LANs?

112 MegaByte/sec

Like the Micosoft windows popup or scp in linux the copy speed is reported in MB (megabyte) per second. Be mindful to not confuse that with network speeds which are reported in bit per second. Let's standardize from here on out that copy speeds should be talked about in bytes per second (MB/s), and not bits per second (Mb/s)

Rationale:

• 1 gbit per second = 1000 megabits (mbps) per second; this is the advertised network speed.
• 1000 Mbps / 8 = 125 MB/s (bytes per second) which is the theoretical max to be expected.
• I observe a solid 112 MB/sec (and have never seen larger) on a 1gpbs wired network; which calculates to 89.6% efficiency and seems reasonable.
  • this is for scp in linux, from a linux to linux system
  • this is also for a samba share from linux which also gets a solid 112 MB/sec reported by the copy popup in microsoft windows
  • I have observed this over the last 10 years from windows 7 and later, now using RHEL/CentOS 7.x, on a wired 1gbps network at work (cisco enterprise switches) where there undoubtedly other traffic on the network and computers I am doing the transfer on. As well as a $10 tigerdirect 4 port wired switch.
• I don't believe SSH results in excessive overhead, because the same ~10% loss (from 125MB/sec down to 112 MB/sec) is also observed going from windows to linux via samba. This ~10% loss (from 125 MB/sec down to 112 MB/sec) is just the TCP/IP overhead in my humble opinion. I have never seen better than 112 MB/sec.
• At the beginning of a transfer I sometimes see as high as 115 MB/sec and it can bounce around a little between 110-115 but it pretty much always averages and settles down on 112 MB/sec.
• This sustained 112 MB/sec transfer speed, between windows and linux over samba as well as scp between linux and linux, can happen even if both systems are processing (such as doing real number crunching work with top showing 60 cores at 100% on a 64 core server).
• note: prior to running rhel 7.x when I was a SLES 11.4 man I consistently observed a scp of ~80 MB/sec and I never found out why, however my SLES 11.4 to windows over samba still did 112 MB/sec.

Also don't forget disk caching. For example if your server has 128 GB RAM and your win10 PC has 32GB of RAM and you just 7-zip tar a single file that is 10 GB, that 10GB single tar file can all be in RAM and not (yet) on the disk. So copy transfer speeds in this case will not be hindered by disk I/O until you run out of ram compared to the file size you are copying and disk caching can't keep up, at which point you will observe the constant 112 MB/sec copy speed drop right down to some lower speed (more so with spinning disks versus SSD's) because of disk i/o. As well as a few other things, independent of the 1 gbps wired network and associated network hardware, can reduce you from that 112 MB/sec transfer speed.

for comparison, the 112 MB/sec I am claiming is 896 Mb/sec.

Worth noting: a huge killer of transfer speed that can reduce it from 112 MB/sec down to kilobytes a second is copying a folder with many subfolders and [small] files so don't ever do that it way if you want transfer speed. In such a case it's best to first tar or iso it all up into one file, however large it may be, and transfer just that one thing and extract it at the destination.

update 2022-01-20:

• the Ciphers and the MACs in /etc/ssh/sshd_config can play an important part in ssh scp speed.
• I know this testing scp on a 100gbps HDR infiniband network, which is 100x faster than 1gbps and observed the following between two RHEL 7.9 servers:
  • Using Ciphers aes256-ctr,aes192-ctr,aes128-ctr and MACs hmac-sha2-512,hmac-sha2-256 which I believe to be the most secure but not the best performance-wise
    • still get 112 MB/sec on 1gbps LAN, so LAN is the weak link not the 2.9ghz xeon cpu
    • on HDR infiniband I thought I would get close to 112 x 100 = 11,200 MB/sec but I maxed out around 270 MB/sec, so it seemed the 2.9ghz cpu was not able to encrypt fast enough to feed the infiniband network
    • this was setting Ciphers and MACs to one specific one each when testing, so I know it was using that one. Otherwise I believe it chooses the first one specified and if that doesn't work then will try the next cipher or mac specified.
    • changing Ciphers and MACs in sshd_config (sorry I forget to which) the max scp speed I ever got over 100gbps HDR infiniband was ~700 MB/sec. I attribute this 700 MB/sec and nothing close to 11,200 MB/sec being the 2.9 ghz cpu not able to crunch fast enough to do the chosen SSH encryption ; weak link is the cpu in this case not the HDR infiniband network.
    • certain Ciphers choices resulted in as low as 60 MB/sec scp speeds even on 100gbps HDR infiniband.

Answer 3

In addition to the other answers:

• check that your wired connections are all full duplex (from the first machine until the destination machine),
• and ensure that nothing else is using the 1gps link other than : your scp, and the negotiating and return packets (acks, etc)
• and that the encryption you use for the scp is not overwhelming either ends CPU capabilities
• that you have large tcp windows (and everything else your hops can handle) to allow more data packets and less tcp overhead paquets
• you may try compressing on the fly, if the file isn't compressed yet and if, again, it doesn't overwhelm either CPUs

If everything is completely in your favor you should approach 70-80pct of 1gbps (ie, 80 pct of 1024 megabit per seconds, ie 0.8*1024/8 megabytes per seconds) , but it could be at the detriment of other types of connections connections (low packet sizes, or needing more responsivness and less latency)

Lastly, never underestimate the bandwidth of carrying a USB key (or drive) over... (by foot, which is also a health bonus)

Answer 4

38MB/s is equal to 304 Mbps which is a pretty good transfer speed on a 1Gbps Link ,

-multiple factors involved if you are experiencing slowness in transfer speed :

1- throughput of the server itself.

2- i/o performance on the server [use #iostat -xm 1] to check if any of the disks are >90% utilized with reads / writes.

3- the L2 Device in between could cause such slowness [try direct connection to avoid this].