9

On my wired LAN, with 1GBit/s devices, I have two Linux machines (One Haswell, One Skylake Xeon) and when I do a secure copy of a large file, I see 38MB/s.

Seeing that this is 3 times below the 1000Mbit/s spec, I wonder if this performance is as expected?

Both machines use SSD for storage, both run 64bit Ubuntu.

During the transfer, both machine have approximately one core at 30% load.

The router that sits between the machines is a TP-Link Archer C7 AC1750. Both machines have Intel(R) Gigabit Ethernet Network devices that are in Full Duplex mode.

What is a normal scp transfer speed on 1Gbit LANs?

UPDATE

  • Using /dev/zero to rule out disk IO yielded the same results.
  • Using nc yielded slightly higher: 41MiB/s.
  • Paradoxically, UDP nc was slower than TCP nc, at 38MiB/s?
  • Switching to crossover cable: 112MB/s for scp.

CONCLUSION

The TP-Link router in between was the weak link in the network, and could not keep up.

  • 3
    both machine have approximately a 30% CPU load - are you sure one of your cores on either of the PCs hasn't maxed out and you're seeing average CPU load across all cores? Run top, press 1 and make sure you're reporting proper values. – Artem S. Tashkinov Sep 19 at 10:13
3

It does seem slow from a theoretical stand point although I've not seen any transfers much quicker practically on home hardware.

Some experiments you might like to try to rule out possible limiting factors:

  • Assess your raw SSH speed by copying from /dev/zero to /dev/null. This rules out a HD bottle neck.
    ssh remote_host cat /dev/zero | pv > /dev/null
    
  • Check other unencrypted protocols such as HTTP. HTTP actually sends files as with nothing but a header. Sending large files over HTTP is a reasonable measure of TCP speeds.
  • Check that you are not forcing traffic through the router but only it's ethernet switch. For example if your machine has a public IP and local IP, then scp to/from the local IP. This is because home routers often have to process WAN traffic through their CPU which creates a bottle neck. Even if both machines are on the LAN, using the public IP can force packets through the CPU as if it was going to the WAN.
  • Similarly I would use IPv4. Some home routers have a weird behaviour with IPv6 where they ask all local traffic to be forwarded to the router.
  • If at all possible try with a gigabit crossover cable instead of a router. This should rule out router.
| improve this answer | |
  • 2
    One could also compare to iperf results. – Carsten S Sep 19 at 9:53
  • 3
    I think that netcat (nc) is best solution to testing "raw" file transfer speed as it is very simple to use and transmits directly over TCP without any wrapping: nc -l 5678 > file_name on listening end and nc recievers_ip 5678 < file_name on sender. – val says Reinstate Monica Sep 19 at 15:44
  • @valsaysReinstateMonica I'm not against that view however the KB of HTTP headers on top of a 100MB+ file sufficiently large to test a GB LAN is trivially insignificant. The commands you've listed there presumably have no output and would need to be piped through PV to measure the speed, am I right? – Philip Couling Sep 20 at 22:57
2

In addition to the other answers:

  • check that your wired connections are all full duplex (from the first machine until the destination machine),
  • and ensure that nothing else is using the 1gps link other than : your scp, and the negotiating and return packets (acks, etc)
  • and that the encryption you use for the scp is not overwhelming either ends CPU capabilities
  • that you have large tcp windows (and everything else your hops can handle) to allow more data packets and less tcp overhead paquets
  • you may try compressing on the fly, if the file isn't compressed yet and if, again, it doesn't overwhelm either CPUs

If everything is completely in your favor you should approach 70-80pct of 1gbps (ie, 80 pct of 1024 megabit per seconds, ie 0.8*1024/8 megabytes per seconds) , but it could be at the detriment of other types of connections connections (low packet sizes, or needing more responsivness and less latency)

Lastly, never underestimate the bandwidth of carrying a USB key (or drive) over... (by foot, which is also a health bonus)

| improve this answer | |
  • 1 gigabit/second should be 1,000 megabit/second above. Only the disk part of the transfer should be using mebibytes (1,024). – smitelli Sep 19 at 13:19
  • 1
    As smitelli says speeds measured in bytes are often factor of 1024. So 1 MBps is 1024*1024 bytes per second. However speeds in bits are always a factor of 1000 because it is exactly the same as bandwidth in Hz. So 1 Mbps is 1000 * 1000 bits per second and requires 1MHz bandwidth. – Philip Couling Sep 19 at 14:33
0

38MB/s is equal to 304 Mbps which is a pretty good transfer speed on a 1Gbps Link ,

-multiple factors involved if you are experiencing slowness in transfer speed :

1- throughput of the server itself.

2- i/o performance on the server [use #iostat -xm 1] to check if any of the disks are >90% utilized with reads / writes.

3- the L2 Device in between could cause such slowness [try direct connection to avoid this].

| improve this answer | |
  • Thanks. Good to know it is not abnormally low. "16%" util in iostat, is the highest I see during a transfer. – Bram Sep 18 at 21:35
  • 4
    "304 Mbps which is a pretty good transfer speed on a 1Gbps Link" Is it? Really? – user414777 Sep 19 at 2:21
  • 2
    If you think 304 Mbps is "pretty good" then I have no idea what kind of hardware you're used to. My crappy 10 year old (yes, literally) home desktop does >100 MB/s over scp to my other crappy 10 year old desktop through two consumer grade switches. Anything else and I would have had to start debugging what's wrong. – pipe Sep 19 at 10:13
  • pipe and the other guy .... we are talking about transferring from one VM to another on a wired lan .. disk speed will get involved and many other parameters ... how ever from my own experience you two nucklheads .. i couldn't get any more that 40 MB/s and this is on the same device of mine on a testing VM on workstation without any wired lan involved – Nour Eddin Sep 19 at 10:31
  • 4
    304Mbps is not a "pretty good" transfer speed over a 1Gbps link. It may be a pretty good rate to get from crappy disks. And please don't call people knuckleheads. – marcelm Sep 19 at 10:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.