29

I'm trying to upload some big files (around 10GB) with a slow upload speed (200kb/s) on a often disconnected SSH connection (due to poor network conditions).

I'm trying to use scp, but if there is a best way over SSH, I'm ok with it.

What is the best way to do it ?

I've tried to split it up in several parts using split, but it's not really efficient as it require a lot of manual work before and after it is transfered.

5
  • 3
    rsync is better. It is restartable - meaning you can continue where you left off if your connection disconnects. You'll probably want to use the --partial flag. And by default it runs over ssh. You could combine with an aggressive compression program like 7z, but you'll need plenty of memory, and it will still take a while. Mar 30, 2015 at 13:39
  • I've tried with --partial, but it just stars over...
    – blue112
    Mar 30, 2015 at 13:40
  • "stars over"? You mean starts over? These are the flags I use: rsync -abvz --partial. Can you show an example session? Mar 30, 2015 at 13:43
  • @FaheemMitha I've tried to reproduce it, but it seems to be working now...
    – blue112
    Mar 30, 2015 at 13:50
  • It's not possible to resume an upload with scp. It's possible to resume an upload with rsync --partial. It's not possible to resume an upload with sftp. For the other direction, i.e. resuming downloads, you can use rsync --partial, sftp reget or sftp get -a.
    – pts
    Mar 30, 2015 at 17:46

1 Answer 1

36

Use rsync with the --partial option

rsync -av --partial sourcedir user@desthost:/destinationdir

The --partial will keep partially transferred files. When you resume the rsync transfer after a ssh broken connection, partially transferred files will start resuming from the point where the ssh connection was lost, and also successfully transferred files will not be transferred again.

Also consider passing in the -z option if you believe the file(s) you are transferring can be compressed significantly; for example, log files comprising of repeated text.

1
  • rsync -LvzP remoteserver:path/to/file localfile Dec 18, 2019 at 16:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .