I'm trying to upload some big files (around 10GB) with a slow upload speed (200kb/s) on a often disconnected SSH connection (due to poor network conditions).
I'm trying to use scp, but if there is a best way over SSH, I'm ok with it.
What is the best way to do it ?
I've tried to split it up in several parts using split, but it's not really efficient as it require a lot of manual work before and after it is transfered.
Use rsync with the --partial option
rsync -av --partial sourcedir user@desthost:/destinationdir
The --partial will keep partially transferred files. When you resume the rsync transfer after a ssh broken connection, partially transferred files will start resuming from the point where the ssh connection was lost, and also successfully transferred files will not be transferred again.
Also consider passing in the -z option if you believe the file(s) you are transferring can be compressed significantly; for example, log files comprising of repeated text.
rsyncis better. It is restartable - meaning you can continue where you left off if your connection disconnects. You'll probably want to use the--partialflag. And by default it runs over ssh. You could combine with an aggressive compression program like 7z, but you'll need plenty of memory, and it will still take a while.rsync -abvz --partial. Can you show an example session?scp. It's possible to resume an upload withrsync --partial. It's not possible to resume an upload withsftp. For the other direction, i.e. resuming downloads, you can usersync --partial,sftp regetorsftp get -a.