23

I'm trying to upload some big files (around 10GB) with a slow upload speed (200kb/s) on a often disconnected SSH connection (due to poor network conditions).

I'm trying to use scp, but if there is a best way over SSH, I'm ok with it.

What is the best way to do it ?

I've tried to split it up in several parts using split, but it's not really efficient as it require a lot of manual work before and after it is transfered.

  • 2
    rsync is better. It is restartable - meaning you can continue where you left off if your connection disconnects. You'll probably want to use the --partial flag. And by default it runs over ssh. You could combine with an aggressive compression program like 7z, but you'll need plenty of memory, and it will still take a while. – Faheem Mitha Mar 30 '15 at 13:39
  • I've tried with --partial, but it just stars over... – blue112 Mar 30 '15 at 13:40
  • "stars over"? You mean starts over? These are the flags I use: rsync -abvz --partial. Can you show an example session? – Faheem Mitha Mar 30 '15 at 13:43
  • @FaheemMitha I've tried to reproduce it, but it seems to be working now... – blue112 Mar 30 '15 at 13:50
  • It's not possible to resume an upload with scp. It's possible to resume an upload with rsync --partial. It's not possible to resume an upload with sftp. For the other direction, i.e. resuming downloads, you can use rsync --partial, sftp reget or sftp get -a. – pts Mar 30 '15 at 17:46
32

Use rsync with the --partial option

rsync -av --partial sourcedir user@desthost:/destinationdir

The --partial will keep partially transferred files. When you resume the rsync transfer after a ssh broken connection, partially transferred files will start resuming from the point where the ssh connection was lost, and also successfully transferred files will not be transferred again.

Also consider passing in the -z option if you believe the file(s) you are transferring can be compressed significantly; for example, log files comprising of repeated text.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.