9

On my wired LAN, with 1GBit/s devices, I have two Linux machines (One Haswell, One Skylake Xeon) and when I do a secure copy of a large file, I see 38MB/s.

Seeing that this is 3 times below the 1000Mbit/s spec, I wonder if this performance is as expected?

Both machines use SSD for storage, both run 64bit Ubuntu.

During the transfer, both machine have approximately one core at 30% load.

The router that sits between the machines is a TP-Link Archer C7 AC1750. Both machines have Intel(R) Gigabit Ethernet Network devices that are in Full Duplex mode.

What is a normal scp transfer speed on 1Gbit LANs?

UPDATE

  • Using /dev/zero to rule out disk IO yielded the same results.
  • Using nc yielded slightly higher: 41MiB/s.
  • Paradoxically, UDP nc was slower than TCP nc, at 38MiB/s?
  • Switching to crossover cable: 112MB/s for scp.

CONCLUSION

The TP-Link router in between was the weak link in the network, and could not keep up.

1
  • 3
    both machine have approximately a 30% CPU load - are you sure one of your cores on either of the PCs hasn't maxed out and you're seeing average CPU load across all cores? Run top, press 1 and make sure you're reporting proper values. – Artem S. Tashkinov Sep 19 '20 at 10:13
3

It does seem slow from a theoretical stand point although I've not seen any transfers much quicker practically on home hardware.

Some experiments you might like to try to rule out possible limiting factors:

  • Assess your raw SSH speed by copying from /dev/zero to /dev/null. This rules out a HD bottle neck.
    ssh remote_host cat /dev/zero | pv > /dev/null
    
  • Check other unencrypted protocols such as HTTP. HTTP actually sends files as with nothing but a header. Sending large files over HTTP is a reasonable measure of TCP speeds.
  • Check that you are not forcing traffic through the router but only it's ethernet switch. For example if your machine has a public IP and local IP, then scp to/from the local IP. This is because home routers often have to process WAN traffic through their CPU which creates a bottle neck. Even if both machines are on the LAN, using the public IP can force packets through the CPU as if it was going to the WAN.
  • Similarly I would use IPv4. Some home routers have a weird behaviour with IPv6 where they ask all local traffic to be forwarded to the router.
  • If at all possible try with a gigabit crossover cable instead of a router. This should rule out router.
3
  • 2
    One could also compare to iperf results. – Carsten S Sep 19 '20 at 9:53
  • 3
    I think that netcat (nc) is best solution to testing "raw" file transfer speed as it is very simple to use and transmits directly over TCP without any wrapping: nc -l 5678 > file_name on listening end and nc recievers_ip 5678 < file_name on sender. – val is still with Monica Sep 19 '20 at 15:44
  • @valsaysReinstateMonica I'm not against that view however the KB of HTTP headers on top of a 100MB+ file sufficiently large to test a GB LAN is trivially insignificant. The commands you've listed there presumably have no output and would need to be piped through PV to measure the speed, am I right? – Philip Couling Sep 20 '20 at 22:57
2

In addition to the other answers:

  • check that your wired connections are all full duplex (from the first machine until the destination machine),
  • and ensure that nothing else is using the 1gps link other than : your scp, and the negotiating and return packets (acks, etc)
  • and that the encryption you use for the scp is not overwhelming either ends CPU capabilities
  • that you have large tcp windows (and everything else your hops can handle) to allow more data packets and less tcp overhead paquets
  • you may try compressing on the fly, if the file isn't compressed yet and if, again, it doesn't overwhelm either CPUs

If everything is completely in your favor you should approach 70-80pct of 1gbps (ie, 80 pct of 1024 megabit per seconds, ie 0.8*1024/8 megabytes per seconds) , but it could be at the detriment of other types of connections connections (low packet sizes, or needing more responsivness and less latency)

Lastly, never underestimate the bandwidth of carrying a USB key (or drive) over... (by foot, which is also a health bonus)

2
  • 1 gigabit/second should be 1,000 megabit/second above. Only the disk part of the transfer should be using mebibytes (1,024). – smitelli Sep 19 '20 at 13:19
  • 1
    As smitelli says speeds measured in bytes are often factor of 1024. So 1 MBps is 1024*1024 bytes per second. However speeds in bits are always a factor of 1000 because it is exactly the same as bandwidth in Hz. So 1 Mbps is 1000 * 1000 bits per second and requires 1MHz bandwidth. – Philip Couling Sep 19 '20 at 14:33
0

38MB/s is equal to 304 Mbps which is a pretty good transfer speed on a 1Gbps Link ,

-multiple factors involved if you are experiencing slowness in transfer speed :

1- throughput of the server itself.

2- i/o performance on the server [use #iostat -xm 1] to check if any of the disks are >90% utilized with reads / writes.

3- the L2 Device in between could cause such slowness [try direct connection to avoid this].

6
  • Thanks. Good to know it is not abnormally low. "16%" util in iostat, is the highest I see during a transfer. – Bram Sep 18 '20 at 21:35
  • 4
    "304 Mbps which is a pretty good transfer speed on a 1Gbps Link" Is it? Really? – user414777 Sep 19 '20 at 2:21
  • 2
    If you think 304 Mbps is "pretty good" then I have no idea what kind of hardware you're used to. My crappy 10 year old (yes, literally) home desktop does >100 MB/s over scp to my other crappy 10 year old desktop through two consumer grade switches. Anything else and I would have had to start debugging what's wrong. – pipe Sep 19 '20 at 10:13
  • pipe and the other guy .... we are talking about transferring from one VM to another on a wired lan .. disk speed will get involved and many other parameters ... how ever from my own experience you two nucklheads .. i couldn't get any more that 40 MB/s and this is on the same device of mine on a testing VM on workstation without any wired lan involved – Nour Eddin Sep 19 '20 at 10:31
  • 5
    304Mbps is not a "pretty good" transfer speed over a 1Gbps link. It may be a pretty good rate to get from crappy disks. And please don't call people knuckleheads. – marcelm Sep 19 '20 at 10:54
0

What is a normal scp transfer speed on 1Gbit LANs?

112 MegaByte/sec

Like the Micosoft windows popup or scp in linux the copy speed is reported in MB (megabyte) per second. Be mindful to not confuse that with network speeds which are reported in bit per second. Let's standardize from here on out that copy speeds should be talked about in bytes per second (MB/s), and not bits per second (Mb/s)

Rationale:

  • 1 gbit per second = 1000 megabits (mbps) per second; this is the advertised network speed.
  • 1000 mbps / 8 = 125 MB/s (bytes per second) which is the theoretical max to be expected.
  • I observe a solid 112 MB/sec (and have never seen larger) on a 1gpbs wired network; which calculates to 89.6% efficiency and seems reasonable.
    • this is for scp in linux, from a linux to linux system
    • this is also for a samba share from linux which also gets a solid 112 MB/sec reported by the copy popup in microsoft windows
    • I have observed this over the last 10 years from windows 7 and later, now using RHEL/CentOS 7.x, on a wired 1gbps network at work (cisco enterprise switches) where there undoubtedly other traffic on the network and computers I am doing the transfer on. As well as a $10 tigerdirect 4 port wired switch.
  • I don't believe SSH results in excessive overhead, because the same ~10% loss (from 125MB/sec down to 112 MB/sec) is also observed going from windows to linux via samba. This ~10% loss (from 125 MB/sec down to 112 MB/sec) is just the TCP/IP overhead in my humble opinion. I have never seen better than 112 MB/sec.
  • At the beginning of a transfer I sometimes see as high as 115 MB/sec and it can bounce around a little between 110-115 but it pretty much always averages and settles down on 112 MB/sec.
  • This sustained 112 MB/sec transfer speed, between windows and linux over samba as well as scp between linux and linux, can happen even if both systems are processing (such as doing real number crunching work with top showing 60 cores at 100% on a 64 core server).
  • note: prior to running rhel 7.x when I was a SLES 11.4 man I consistently observed a scp of ~80 MB/sec and I never found out why, however my SLES 11.4 to windows over samba still did 112 MB/sec.

Also don't forget disk caching. For example if your server has 128 GB RAM and your win10 PC has 32GB of RAM and you just 7-zip tar a single file that is 10 GB, that 10GB single tar file can all be in RAM and not (yet) on the disk. So copy transfer speeds in this case will not be hindered by disk I/O until you run out of ram compared to the file size you are copying and disk caching can't keep up, at which point you will observe the constant 112 MB/sec copy speed drop right down to some lower speed (more so with spinning disks versus SSD's) because of disk i/o. As well as a few other things, independent of the 1 gbps wired network and associated network hardware, can reduce you from that 112 MB/sec transfer speed.

for comparison, the 112 MB/sec I am claiming is 896 Mb/sec.

Worth noting: a huge killer of transfer speed that can reduce it from 112 MB/sec down to kilobytes a second is copying a folder with many subfolders and [small] files so don't ever do that it way if you want transfer speed. In such a case it's best to first tar or iso it all up into one file, however large it may be, and transfer just that one thing and extract it at the destination.

enter image description here

enter image description here

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.