One Rsync Client takes well over 10 hours

Question

rsync version 3.1.3-6 protocal version 31 cifs-utils/stable 2:6.8-2 amb64 nfs-common/stable 1:1.3.4-2.5 Debian Linux 10 Buster FreeNAS 11.3-U3.2 (IxSystems FreeNAS Mini)

Scenario where I am using a small virtual Debian Linux server to backup some of my remote facilities. I used the Debian server - because I could not get FreeNAS (FreeBSD) to mount cifs shares consistently without much fuss, i'm a little more familiar with Linux and don't like messing under FreeNAS's hood if I can avoid it - however I could get Debain to mount cifs consistently and reliably as well as NFS to mount the FreeNAS server - which has been working very well for months. So my little Debian backup server is the go between for the FreeNAS server and my Client computers.

Debian - Mount's and NFS share to FreeNAS on boot - permanent - haven't experienced issues. Debian - run's small batch script every 6 hours to mount the target client Windows PC's and then rsync their data from the mounted cifs share to the mounted nfs share.

I am thinking this may be problem - found this post where a users states

As far as rsync is concerned you're copying between two local file trees, so it disables most of its optimisations (including its delta algorithm for which it is famous).

So my question would be, is there a way to force the optimizations anyway, even though they are considered to be two local file systems? In that post, they were trying to delete, which I am not - mine is all about backing up all the users hoarding.

Answer 1

You can force an optimisation with the --no-whole-file flag, but it is highly unlikely to do anything other than slow down the transfer process.

Before you even consider this option, ensure you are using the --archive (-a) or --times (-t) flag so that file modification times are retained. In conjunction with the file size this will help rsync decide whether its "quick check" is sufficient to skip a file rather than recopying it.

Here's why --no-whole-file is such a bad idea.

The delta algorithm checks blocks of a file to see which ones need to be transmitted. For a small change it may be that only one block has to change, and that can be very attractive. However, in order to determine which blocks have changed, both the source file and the destination file have to be read in their entirety. When you have two different systems this can be done in parallel using two independent processor/disk combinations, and the source file has to be read anyway, so the only hit is reading the target file on the destination system. When you have two local filesystems, though, both files still need to be read but processor and disk contention comes in to play.

In a worst case it means that to write one changed block, both files have to be read from beginning to end. But by this point you might as well have just read one and written the other, as you've still got to rewrite some or all of the target file anyway.