rsync: continue or in random order

Question

I use rsync over tor. It works for a while, then it drops the connection. That in itself is not a big problem and I suppose it is not entirely unexpected when running long sessions over tor that takes many minutes to complete.

When I restart rsync it goes through the files in the same order as last time. That is a problem because some of these files change often (think of /var/spool of a busy mail server and you get the idea). This means that the files it gets to first get updated often, while files it never gets around to are never transferred.

What I would prefer is to have rsync either continue where it left off or at least transfer the files in random order.

Is there a general way of doing that?

Answer 1

The solution became this:

find . -maxdepth 4 -depth -type d | shuf | torsocks parallel rsync --delete -zHax --inplace --rsync-path='mkdir -p {} && rsync' ~/{}/ qnek3r4buxhaa6g3.onion:./{}/

Not exactly pretty, but it seems to do the job.

Answer 2

Use the switch --partial. This makes rsync carry on from where it left off.

Answer 3

There are a couple of options available. You can use filter rules on some of the rsync runs so that the busiest files are ignored, allowing less busy files to be transferred. If the list of busy files is too long to be easily assembled into a set of filter rules, you can use --modify-window with a large value on some of the rsync runs. This too will have the effect of ignoring recently modified files and transferring files that were modified outside the modification window.