123

I'm trying to copy a batch of files with scp but it is very slow. This is an example with 10 files:

$ time scp cap_* user@host:~/dir
cap_20151023T113018_704979707.png    100%  413KB 413.2KB/s   00:00    
cap_20151023T113019_999990226.png    100%  413KB 412.6KB/s   00:00    
cap_20151023T113020_649251955.png    100%  417KB 416.8KB/s   00:00    
cap_20151023T113021_284028464.png    100%  417KB 416.8KB/s   00:00    
cap_20151023T113021_927950468.png    100%  413KB 413.0KB/s   00:00    
cap_20151023T113022_567641507.png    100%  413KB 413.1KB/s   00:00    
cap_20151023T113023_203534753.png    100%  414KB 413.5KB/s   00:00    
cap_20151023T113023_855350640.png    100%  412KB 411.7KB/s   00:00    
cap_20151023T113024_496387641.png    100%  412KB 412.3KB/s   00:00    
cap_20151023T113025_138012848.png    100%  414KB 413.8KB/s   00:00    
cap_20151023T113025_778042791.png    100%  413KB 413.4KB/s   00:00    

real    0m43.932s
user    0m0.074s
sys 0m0.030s

The strange thing is that the transfer rate is about 413KB/s and the file size is about 413KB so really it should transfer one file per second, however it's taking about 4.3 seconds per file.

Any idea where this overhead comes from, and is there any way to make it faster?

Improve this question
7
  • 4
    What speed do you expect (i.e., is there another protocol that shows higher transfer speeds between the same two machines)? What happens when you scp a much larger file (perhaps the concatenation of all you 413KB files)?
    – dhag
    Oct 23, 2015 at 13:47
  • 9
    It looks like the remote system may be trying to resolve the client IP address to a name, and you're having to wait for a timeout before the session proceeds. You could investigate fixing that (e.g. add your IP address to the destination's /etc/hosts file).
    – wurtel
    Oct 23, 2015 at 14:35
  • 9
    It's worth mentioning that the -C flag enables compression during transfer. Although your problem seems to be overhead starting transfers, compression is basically "free" and almost always helps.
    – Sam
    Oct 24, 2015 at 2:53
  • 1
    @wurtel: I don't see what you're seeing, all I see are times. There should only be a single reverse DNS call needed anyway.
    – President James K. Polk
    Oct 24, 2015 at 12:36
  • 1
    Are you relying on SCP for security or only for remote copying?
    – Freiheit
    Oct 26, 2015 at 13:37

8 Answers 8

Reset to default
122

You could use rsync (over ssh), which uses a single connection to transfer all the source files.

rsync -avP cap_* user@host:dir

If you don't have rsync (and why not!?) you can use tar with ssh like this, which avoids creating a temporary file (these two alternatives are equivalent):

tar czf - cap_* | ssh user@host tar xvzfC - dir
tar cf - cap_* | gzip | ssh user@host 'cd dir && gzip -d | tar xvf -'

The rsync is to be preferred, all other things being equal, because it's restartable in the event of an interruption.

Improve this answer
14
  • 12
    Are you saying a single scp invocation wouldn't use a single connection to transfer all files?
    – user
    Oct 23, 2015 at 21:11
  • 2
    In the tarpipe case, there's no need for the f - on each side, since tar outputs to/reads from stdout/stdin by default. So tar cz cap_* | ssh user@host tar xvzC dir would do it.
    – tremby
    Oct 24, 2015 at 1:41
  • 2
    @tremby not necessarily. tar can be compiled with different default values (see tar --show-defaults if you're using GNU tar, or /etc/default/tar otherwise, and in both cases don't forget the TAPE environment variable)
    – Chris Davies
    Oct 24, 2015 at 11:16
  • 1
    @MichaelKjörling initially I had assumed that scp would create a new connection for each file, but on recollection - and after double-checking with tshark - I realised that I was incorrect. At this point I'm no longer sure why the OP's scp should be taking such a long time per file.
    – Chris Davies
    Oct 24, 2015 at 11:19
  • 3
    This is so much faster its crazy... thanks so much
    – codenamejames
    Sep 3, 2017 at 22:35
31

@wurtel's comment is probably correct: there's a lot of overhead establishing each connection. If you can fix that you'll get faster transfers (and if you can't, just use @roaima's rsync workaround). I did an experiment transferring similar-sized files (head -c 417K /dev/urandom > foo.1 and made some copies of that file) to a host that takes a while to connect (HOST4) and one that responds very quickly (HOST1):

$ time ssh $HOST1 echo


real    0m0.146s
user    0m0.016s
sys     0m0.008s
$ time scp * $HOST1:
foo.1                                         100%  417KB 417.0KB/s   00:00    
foo.2                                         100%  417KB 417.0KB/s   00:00    
foo.3                                         100%  417KB 417.0KB/s   00:00    
foo.4                                         100%  417KB 417.0KB/s   00:00    
foo.5                                         100%  417KB 417.0KB/s   00:00    

real    0m0.337s
user    0m0.032s
sys     0m0.016s
$ time ssh $HOST4 echo


real    0m1.369s
user    0m0.020s
sys     0m0.016s
$ time scp * $HOST4:
foo.1                                         100%  417KB 417.0KB/s   00:00    
foo.2                                         100%  417KB 417.0KB/s   00:00    
foo.3                                         100%  417KB 417.0KB/s   00:00    
foo.4                                         100%  417KB 417.0KB/s   00:00    
foo.5                                         100%  417KB 417.0KB/s   00:00    

real    0m6.489s
user    0m0.052s
sys     0m0.020s
$
Improve this answer
2
  • 1
    Thanks, that's very interesting. The scp output is kind of broken if it shows the same time even though it's completely different from one host to another. They should probably include the connection time in the total time.
    – laurent
    Feb 4, 2016 at 19:39
  • 4
    So your hypothesis is it makes a new connection once for each file?
    – rogerdpack
    Sep 27, 2018 at 21:59
20

It's the negotiation of the transfer that takes time. In general, operations on n files of b bytes each takes much, much longer than a single operation on a single file of n * b bytes. This is also true e.g. for disk I/O.

If you look carefully you'll see that the transfer rate in this case is size_of_the_file/secs.

To transfer files more efficiently, bundle them together with tar, then transfer the tarball:

tar cvf myarchive.tar cap_20151023T*.png

or, if you also want to compress the archive,

tar cvzf myarchive.tar.gz myfile*

Whether to compress or not depends on the file contents, eg. if they're JPEGs or PNGs, compression won't have any effect.

Improve this answer
6
  • PNGs use deflate, and gzipping them is pointless too.
    – Mingye Wang
    Oct 23, 2015 at 15:06
  • I'd say that because compressing the tar does not have negative effects when the files can't be compressed further it's a good practice to just put -z
    – Centimane
    Oct 23, 2015 at 15:49
  • 1
    @Dave if they can't be compressed, or the network is fast, it will slow things down.
    – Davidmh
    Oct 23, 2015 at 16:51
  • @Davidmh would this be by a significant amount though? I would think compressing an already compressed file would be fairly quick as it would really just look over what it could compress and find that it is nothing. Depends I guess if tar normally does a second pass for compression or if it would be compressing and archiving at the same time
    – Centimane
    Oct 23, 2015 at 18:05
  • 5
    @Dave in my case (data on a modern 7000 rpm HD, high end CPU, very fast network, not bragging at all), tar without compression is purely IO bound, but with -z is CPU bound, and much slower. gzip will always try to compress, hence the slowdown; after all, you can't tell if a string of bytes is compressible until you have tried to compress it. In my set up, even when transferring plain text files, rsync without compression is the fastest by a factor of 2-3 compared with the lightest compression. Of course, YMMV.
    – Davidmh
    Oct 23, 2015 at 18:32
9

I've used the technique described here (archived) which uses parallel gzip and netcat to quickly compress and copy data.

It boils down to:

# SOURCE: 
> tar -cf - /u02/databases/mydb/data_file-1.dbf | pigz | nc -l 8888

# TARGET:
> nc <source host> 8888 | pigz -d | tar xf - -C /

This uses tar to gather up the file or files. Then uses pigz to get many cpu threads to compress and send the file, the network transmission is using netcat. On the receiving side, netcat listens then uncompresses (in parallel) and untars.

Improve this answer
3
  • 5
    nc is not encrypted. Add some ssh -D magic maybe?
    – Mingye Wang
    Oct 23, 2015 at 23:04
  • this is actually pretty brilliant
    – Jabran Saeed
    Mar 24, 2016 at 10:10
  • 1
    Better than disabling encryption, the linked article concludes "the magic bullet is compression", so just enable compression! add Compression=yes on either your /.ssh/config or via -o to scp.
    – gcb
    Mar 20, 2021 at 18:13
8

Another reason that scp is slower than it should be, especially on high bandwidth networks, is that it has statically defined internal flow control buffers which end up becoming network performance bottlenecks.

HPN-SSH is a patched version of OpenSSH which increases the size of these buffers. It makes a massive difference to scp transfer speed (see the charts on the site, but I also speak from personal experience). Of course, to get the benefits you need to install HPN-SSH on all your hosts but it's well worth it if you regularly need to transfer large files around.

Improve this answer
7

Just had this issue doing a site-to-site transfer of a large mp4 file via scp. Was getting ~250KB/s. After disabling UDP flood protection (FP) on the destination firewall, the transfer rate increased to 6.5MB/s. When turning FP back on, the rate dropped back to ~250KB/s.

Sender: cygwin, Receiver: Fedora 20, Firewall Sophos UTM.

What does SSH use UDP for? @ superuser.com -- It doesn't directly from what I read.

In reviewing the firewall log, flood detection was occurring on both source & dest ports 4500 over the public IP addresses, not the private site-to-site internal VPN addresses. So it seems my issue is likely a NAT Traversal situation where the scp TCP data is ultimately encrypted and encapsulated in ESP & UDP packets, and consequently subject to FP. To remove scp from the equation, I ran a Windows file copy operation across the VPN and noticed similar performance to scp with and without FP enabled. Also ran an iperf test over TCP and noticed 2Mbits/sec with FP, and 55Mbits/sec without.

How Does NAT-T work with IPSec? @ cisco.com

Improve this answer
3

Since this question is not that old and no one else referred to this solution I think it is appropriate since it pushes the bandwidth to the max limit (10MiB/s in my case) unlike scp that is at arount 250kb/s so that answers your question.

Actually the same 250kb/s with rsync - at least with the port specifier rclone -Avvp cap_* -e "ssh -p 1087 -i id_rsa" user@host:~/dir

Quoting a post to the openssh-unix-dev mailing list:

The scp protocol is outdated, inflexible and not readily fixed. Its authors recommend the use of more modern protocols like sftp and rsync for file transfer instead

The same syntax applies to sftp so instead of scp text.txt user@host it is now sftp text.txt user@host(usage examples scp interchangable with sftp)

Also the recent version of OpenSSH should activate the daemon - at least in my case on an arch linux server but you might have to install an sftp package on other distributions.

One more working example with an ssh encryption file flag (id_rsa) and a non standard ssh port 1087 instead of 22 to safe you time fiddeling with the syntax:

sftp -P 1087 -i id_rsa user@server:/home/user/Downloads/Video/*/*.mp4 /home/user/Videos/

Also your sftp might be limited to 800kb/s or ~1 Mbit/s. You can check this with:

# sysctl -a | grep net.*rmem

and you can change the limits e.g. like this if they are too slow:

   # sysctl -w net.ipv4.tcp_rmem='40960 873800 62914560'

   # sysctl -w net.core.rmem_max=8388608
Improve this answer
1
2

You can make your server into an online website

$ sudo apt-get install apache2
# open port 80 on your server 
# copy your files to the server at /var/www/html/
# if you can't copy or symlink them, use:
$ sudo chown $USER /var/www/html/

and download your files using wget or curl...

$ wget  http://40.86.167.128/video.mp4

For me, I got 100% speed on wifi 4mbit/s, which means my speed is 4mbit/s
and 25% for wifi 100mbit/s, which means 25mbit/s, not 100mbits
but with rsync or scp, I got 1mbit/s for wifi 4mbit/s

Improve this answer
3
  • I'm un-deleting this as it attempts to answer the question. Whether a second intermediate copy would be worth the HTTP savings isn't clear to me, but it's worth testing.
    – Jeff Schaller ♦
    May 21, 2022 at 23:56
  • In my case, the file that I wanted to download via scp was already in /var/www/html and being served by a web server (nginx), so I could use curl instead of scp as you suggest and it's much faster. My only concern is that the file (which contains an IP address) should perhaps not be available publicly, so it should be protected somehow, or I should go back to a solution that does not require it to be on a web server... But if the file in question is not a secret, this works very well.
    – drkvogel
    Dec 30, 2023 at 11:54
  • When we move files to /var/www/html/, they become public by default. You should have experience with nginx to handle private files. I don't know why wget/curl is faster than scp/rsync, but this method could be helpful sometimes.
    – nextloop
    Jan 9 at 18:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .