Why does my scp cap at 8MB over local network transfers?

Question

Is there a reason scp caps at around 8MB when transferring files from one computer to another, over a local network? I'm transferring a couple hundred GB of data (movies and such), and have ran the following command:

scp -rv ../../cygdrive/e/plex [email protected]:/home/caleb/Desktop

This is transferring from my windows box to my ubuntu box. Almost every file caps out at ~8MB/s:

Justified Season 2 Episode 10 - Debts and Accounts.avi                                                                              100%  350MB   7.8MB/s   00:45
Sending file modes: C0050 367218038 Justified Season 2 Episode 11 - Full Commitment.avi
Sink: C0050 367218038 Justified Season 2 Episode 11 - Full Commitment.avi
Justified Season 2 Episode 11 - Full Commitment.avi                                                                                 100%  350MB   7.8MB/s   00:45
Sending file modes: C0050 367451326 Justified Season 2 Episode 12 - Reckoning.avi
Sink: C0050 367451326 Justified Season 2 Episode 12 - Reckoning.avi
Justified Season 2 Episode 12 - Reckoning.avi                                                                                       100%  350MB   7.8MB/s   00:45
Sending file modes: C0050 367779546 Justified Season 2 Episode 13 - Bloody Harlan.avi
Sink: C0050 367779546 Justified Season 2 Episode 13 - Bloody Harlan.avi
Justified Season 2 Episode 13 - Bloody Harlan.avi 

I'm 100% sure this is going over LAN, and both machines are hardwired into a 1000Mb/s (1Gb) switch on my router. I'll rarely see one file jump up to 100MB/s+, but almost all of them are capped as above.

Am I missing something? What other troubleshooting steps can I take?

Answer 1

Firstly you shoud check if it hits the storage performance limit on either Windows or Ubuntu. If E: on Windows is a USB attached drive or memory card, 8MB/s is likely possible.

Otherwise SSH encryption might be bottleneck. Try and see with lightweight cipher like arcfour.

scp -o Cipher=arcfour -rv /cygdrive/e/plex [email protected]:/home/caleb/Desktop

And it could suffer from Cygwin binary's overhead. You might want to use WinSCP to gain better performance.

Alternatively you can set up Samba share on the Ubuntu host. SMB could get closer to the real wire speed on your LAN with less overhead than SCP.

Answer 2

Your actual speed is 8MB/s x 8 bits per byte= 64 Mb/s (megabits per second). That's about 64% of the theoretical maximum. With OS overheads, TCP/IP ACKs, etc. I'm not too surprised. The earlier tips may help you push that up a bit, but you're never going to get above ~12 MB/s unless you go to gigabit Ethernet.

Answer 3

The speed of network links is measured in megabits per second, not megabyte per second. A “100 Mb/s” network (the b should be lowercase) can transmit 100 megabits per second. The next speed up is 10 times as fast and called “1 Gb/s” or “gigabit”.

The raw network speed is faster than the application speed, because there's some overhead for packet headers. The maximum size for an Ethernet packet is 1500 bytes, which results in a 1448-byte TCP packet at best (20 bytes for the IP header and 32 bytes for the TCP header). I don't know exactly how much overhead SSH adds, but it must allow for at least some protocol headers and a MAC (20 or 32 bytes), so at least 48 bytes. That's 1400 bytes out of 1500. Add space for the recipient to send TCP ACK messages back (66 bytes), and we're down to 1334/1500. At 100 Mb/s = 12.5 MB/s, that's 11.1 MB/s of data speed.

11.1 MB/s is faster than 8 MB/s, but not a lot faster. There is some unavoidable overhead caused by Ethernet retransmission, which I didn't take into account above, as well as a small overhead for the router. A maximum achievable rate of 8 MB/s sounds plausible.

Home routers are typically limited to 100 Mb/s. Gigabit routers cost more. The network card on reasonably recent home PCs often supports gigabit networks though. If you get a gigabit-capable Ethernet cable (the cheapest ones are only rated for 100 Mb/s) and connect the two computers directly, you may be able to get a connection that's 10 times faster. You can check what speeds your network card supports by running /sbin/ethtool eth0; look for a “link mode” of “1000baseT” (gigabit) in addition to “100baseT” (100 Mb/s).