Most Efficient way to back up/sync large directories over SSH

Question

I'm looking to back up some large directories on a Dedicated Server to another Server on a daily basis using minimal bandwidth

RSync looks like its up to the task but I'm unsure of the best command switches

I would like to do multiple selected directories to one "backup" directory on another machine over SSH

Any ideas?

Answer 1

I'd like to add that you'd better user --numeric-ids flag to preserve uids/gids of your files so they would not be re-mapped to local users.

If you're going to keep historical backups (say, YYYY-MM-DD daliy directories), option --link-dest=DIR would be of great use.

In case your data can be effectively compressed (source code, plaintext files, logs) it -z can be quite handy.

Otherwise, -aH would cover most of your needs.

Answer 2

Try doing this (from backup server ):

rsync \
    -a \
    --bwlimit=100 \
    remote-server:"/path1 /path2 /path3" /local/backup_$(date +%Y%m%d_%H:%M)

--bwlimit=100 stands for al most 100KB/sec

You can use hard links with cp -al between days, that way you can save a lot of disk space and rsync only changed files !

That's how rsnapshot project works.

Answer 3

rsync is ok, but remember you need to be running rsyncd on one of the machines to take advantage of rsync's change-detection algorithm.

As for the hardlinking sputnick mentions, you might also be interested in fdupes.

Answer 4

Rsync is indeed the right low-level tool to handle this. Rather than worry about rsync options, use a dedicated backup tools, there are plenty. Rsnapshot is a popular choice, see if your requirements (which you aren't very explicit about) meet its capabilities. Duplicity is another backup tool based on the rsync library.