55

I'm trying to copy a batch of files with scp but it is very slow. This is an example with 10 files:

$ time scp cap_* user@host:~/dir
cap_20151023T113018_704979707.png    100%  413KB 413.2KB/s   00:00    
cap_20151023T113019_999990226.png    100%  413KB 412.6KB/s   00:00    
cap_20151023T113020_649251955.png    100%  417KB 416.8KB/s   00:00    
cap_20151023T113021_284028464.png    100%  417KB 416.8KB/s   00:00    
cap_20151023T113021_927950468.png    100%  413KB 413.0KB/s   00:00    
cap_20151023T113022_567641507.png    100%  413KB 413.1KB/s   00:00    
cap_20151023T113023_203534753.png    100%  414KB 413.5KB/s   00:00    
cap_20151023T113023_855350640.png    100%  412KB 411.7KB/s   00:00    
cap_20151023T113024_496387641.png    100%  412KB 412.3KB/s   00:00    
cap_20151023T113025_138012848.png    100%  414KB 413.8KB/s   00:00    
cap_20151023T113025_778042791.png    100%  413KB 413.4KB/s   00:00    

real    0m43.932s
user    0m0.074s
sys 0m0.030s

The strange thing is that the transfer rate is about 413KB/s and the file size is about 413KB so really it should transfer one file per second, however it's taking about 4.3 seconds per file.

Any idea where this overhead comes from, and is there any way to make it faster?

  • 3
    What speed do you expect (i.e., is there another protocol that shows higher transfer speeds between the same two machines)? What happens when you scp a much larger file (perhaps the concatenation of all you 413KB files)? – dhag Oct 23 '15 at 13:47
  • 6
    It looks like the remote system may be trying to resolve the client IP address to a name, and you're having to wait for a timeout before the session proceeds. You could investigate fixing that (e.g. add your IP address to the destination's /etc/hosts file). – wurtel Oct 23 '15 at 14:35
  • 4
    It's worth mentioning that the -C flag enables compression during transfer. Although your problem seems to be overhead starting transfers, compression is basically "free" and almost always helps. – Sam Oct 24 '15 at 2:53
  • @wurtel: I don't see what you're seeing, all I see are times. There should only be a single reverse DNS call needed anyway. – James K Polk Oct 24 '15 at 12:36
  • Are you relying on SCP for security or only for remote copying? – Freiheit Oct 26 '15 at 13:37
16

@wurtel's comment is probably correct: there's a lot of overhead establishing each connection. If you can fix that you'll get faster transfers (and if you can't, just use @roaima's rsync workaround). I did an experiment transferring similar-sized files (head -c 417K /dev/urandom > foo.1 and made some copies of that file) to a host that takes a while to connect (HOST4) and one that responds very quickly (HOST1):

$ time ssh $HOST1 echo


real    0m0.146s
user    0m0.016s
sys     0m0.008s
$ time scp * $HOST1:
foo.1                                         100%  417KB 417.0KB/s   00:00    
foo.2                                         100%  417KB 417.0KB/s   00:00    
foo.3                                         100%  417KB 417.0KB/s   00:00    
foo.4                                         100%  417KB 417.0KB/s   00:00    
foo.5                                         100%  417KB 417.0KB/s   00:00    

real    0m0.337s
user    0m0.032s
sys     0m0.016s
$ time ssh $HOST4 echo


real    0m1.369s
user    0m0.020s
sys     0m0.016s
$ time scp * $HOST4:
foo.1                                         100%  417KB 417.0KB/s   00:00    
foo.2                                         100%  417KB 417.0KB/s   00:00    
foo.3                                         100%  417KB 417.0KB/s   00:00    
foo.4                                         100%  417KB 417.0KB/s   00:00    
foo.5                                         100%  417KB 417.0KB/s   00:00    

real    0m6.489s
user    0m0.052s
sys     0m0.020s
$ 
  • Thanks, that's very interesting. The scp output is kind of broken if it shows the same time even though it's completely different from one host to another. They should probably include the connection time in the total time. – laurent Feb 4 '16 at 19:39
  • 2
    @this.lau_ I think it's correct as-is. It measures the time it takes to actually transfer the file, which is useful information (lets you know about / test the connection speed, for example) that can't be measured with an external tool. But there are a lot of irrelevant variables in connection setup (like reverse host lookup, or the user slowly entering their password and getting it wrong the first time) and you can use external tools to measure them (like I showed!). – drewbenn Feb 4 '16 at 19:54
  • So your hypothesis is it makes a new connection once for each file? – rogerdpack Sep 27 '18 at 21:59
57

You could use rsync (over ssh), which uses a single connection to transfer all the source files.

rsync -avP cap_* user@host:dir

If you don't have rsync (and why not!?) you can use tar with ssh like this, which avoids creating a temporary file:

tar czf - cap_* | ssh user@host tar xvzfC - dir

The rsync is to be preferred, all other things being equal, because it's restartable in the event of an interruption.

  • 6
    Are you saying a single scp invocation wouldn't use a single connection to transfer all files? – a CVn Oct 23 '15 at 21:11
  • 1
    In the tarpipe case, there's no need for the f - on each side, since tar outputs to/reads from stdout/stdin by default. So tar cz cap_* | ssh user@host tar xvzC dir would do it. – tremby Oct 24 '15 at 1:41
  • 1
    @tremby not necessarily. tar can be compiled with different default values (see tar --show-defaults if you're using GNU tar, or /etc/default/tar otherwise, and in both cases don't forget the TAPE environment variable) – roaima Oct 24 '15 at 11:16
  • 1
    @MichaelKjörling initially I had assumed that scp would create a new connection for each file, but on recollection - and after double-checking with tshark - I realised that I was incorrect. At this point I'm no longer sure why the OP's scp should be taking such a long time per file. – roaima Oct 24 '15 at 11:19
  • @roaima, interesting, thanks. I've never noticed stdin/stdout not being default so far. BSD tar on my Mac at work doesn't mention a TAPE env var in its man page, though GNU tar on my Linux machine does. – tremby Oct 26 '15 at 17:37
15

It's the negotiation of the transfer that takes time. In general, operations on n files of b bytes each takes much, much longer than a single operation on a single file of n * b bytes. This is also true e.g. for disk I/O.

If you look carefully you'll see that the transfer rate in this case is size_of_the_file/secs.

To transfer files more efficiently, bundle them together with tar, then transfer the tarball:

tar cvf myarchive.tar cap_20151023T*.png

or, if you also want to compress the archive,

tar cvzf myarchive.tar.gz myfile*

Whether to compress or not depends on the file contents, eg. if they're JPEGs or PNGs, compression won't have any effect.

  • PNGs use deflate, and gzipping them is pointless too. – Arthur2e5 Oct 23 '15 at 15:06
  • I'd say that because compressing the tar does not have negative effects when the files can't be compressed further it's a good practice to just put -z – Centimane Oct 23 '15 at 15:49
  • 1
    @Dave if they can't be compressed, or the network is fast, it will slow things down. – Davidmh Oct 23 '15 at 16:51
  • @Davidmh would this be by a significant amount though? I would think compressing an already compressed file would be fairly quick as it would really just look over what it could compress and find that it is nothing. Depends I guess if tar normally does a second pass for compression or if it would be compressing and archiving at the same time – Centimane Oct 23 '15 at 18:05
  • 3
    @Dave in my case (data on a modern 7000 rpm HD, high end CPU, very fast network, not bragging at all), tar without compression is purely IO bound, but with -z is CPU bound, and much slower. gzip will always try to compress, hence the slowdown; after all, you can't tell if a string of bytes is compressible until you have tried to compress it. In my set up, even when transferring plain text files, rsync without compression is the fastest by a factor of 2-3 compared with the lightest compression. Of course, YMMV. – Davidmh Oct 23 '15 at 18:32
6

Another reason that scp is slower than it should be, especially on high bandwidth networks, is that it has statically defined internal flow control buffers which end up becoming network performance bottlenecks.

HPN-SSH is a patched version of OpenSSH which increases the size of these buffers. It makes a massive difference to scp transfer speed (see the charts on the site, but I also speak from personal experience). Of course, to get the benefits you need to install HPN-SSH on all your hosts but it's well worth it if you regularly need to transfer large files around.

5

I've used the technique described here which uses parallel gzip and netcat to quickly compress and copy data.

It boils down to:

# SOURCE: 
> tar -cf - /u02/databases/mydb/data_file-1.dbf | pigz | nc -l 8888

# TARGET:
> nc <source host> 8888 | pigz -d | tar xf - -C /

This uses tar to gather up the file or files. Then uses pigz to get many cpu threads to compress and send the file, the network transmission is using netcat. On the receiving side, netcat listens then uncompresses (in parallel) and untars.

  • 3
    nc is not encrypted. Add some ssh -D magic maybe? – Arthur2e5 Oct 23 '15 at 23:04
  • this is actually pretty brilliant – Jabran Saeed Mar 24 '16 at 10:10
5

Just had this issue doing a site-to-site transfer of a large mp4 file via scp. Was getting ~250KB/s. After disabling UDP flood protection (FP) on the destination firewall, the transfer rate increased to 6.5MB/s. When turning FP back on, the rate dropped back to ~250KB/s.

Sender: cygwin, Receiver: Fedora 20, Firewall Sophos UTM.

What does SSH use UDP for? @ superuser.com -- It doesn't directly from what I read.

In reviewing the firewall log, flood detection was occurring on both source & dest ports 4500 over the public IP addresses, not the private site-to-site internal VPN addresses. So it seems my issue is likely a NAT Traversal situation where the scp TCP data is ultimately encrypted and encapsulated in ESP & UDP packets, and consequently subject to FP. To remove scp from the equation, I ran a Windows file copy operation across the VPN and noticed similar performance to scp with and without FP enabled. Also ran an iperf test over TCP and noticed 2Mbits/sec with FP, and 55Mbits/sec without.

How Does NAT-T work with IPSec? @ cisco.com

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.