1

I'm copying large files over LAN, and I've noticed that with doing one ssh pipe, I can get around 10MB/s transfer speed. However, if I shoot off four of those at the same time, I get around 40MB/s transfer speeds in total.

So I'd like to split up a large file (eg. 1GB) into parts, and pipe each part on its own, then reassemble on other side. Need to tar/zip before transfer also, and untar/unzip on other side.

How can I do this?

  • is this an option? – A.B. Mar 27 '15 at 17:35
  • Hum, perhaps. Not sure what is the bottleneck, actually. Any clue? – knutole Mar 27 '15 at 17:50
  • 2
    It's doable but it seems a rather horrible workaround. Wouldn't it be better to find out why you are not getting full speed under ssh with one pipe? Which os? Which ssh? – gogoud Mar 27 '15 at 18:44
  • 1
    is there a reason why you don't have/want more recent versions? My ubuntu 14.04.2 has OpenSSH_6.6.1p1 Ubuntu-2ubuntu2, OpenSSL 1.0.1f 6 Jan 2014? The latest openssh is 6.8. Some (old) speed comparison info here - best results were tar over ssh using arcfour128 (a low security cipher, disabled by default in openssh 6.7+) or umac-64. Also consider trying HPN-SSH - patch files for openssh 6.6 and earlier are available on sourceforce. – gogoud Mar 28 '15 at 6:41
  • 1
    A Ubuntu ppa for Openssh-hpn, for easy installation, is here. – gogoud Mar 28 '15 at 6:48
2

Ended up with HPN-SSHand pigz.

tar -cf - -C [RELATIVEFOLDER] [FILENAME] | pigz | ssh px "pigz -d | tar xf - -C [REMOTEFOLDER]"

Improvement by power of ten.

For reference, installing HPN-SSH and pigz on Ubuntu 14.04 is easy as:

# hpn-patched ssh from ppa
sudo apt-get install python-software-properties
sudo add-apt-repository ppa:w-rouesnel/openssh-hpn
sudo apt-get update -y  
sudo apt-get install openssh-server
ssh -V # should have 'hpn' in it somewhere

# pigz
sudo apt-get install pigz
  • I'd be interested if you know how much of this 10x speed improvement (to 100MB/sec i.e. network limited?) is because of using HPN-SSH (as I suggested) in place of standard openssh? – gogoud Mar 28 '15 at 17:21
  • @gogoud basically all of it, as the bottleneck was SSH encrypting on one CPU only. Never going back! :) – knutole Mar 29 '15 at 14:37
0

Well there is zipsplit(1) and split(1), that you could used to split up the large files before using say scp(1) to copy them over. Or you could split the output into named pipes and then cat the named pipes to ssh, if you wish a pipelined effect.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.