0

My understanding of rsync (and it might be completely wrong) is that it would first try to read the files (source and destination if it exists) to compare them before attempting to write something on the destination (which I would expect to be slower than reading). I am running rsync to sync up a backup of my home and instead of seeing a lot of reading on source and destination disks (right after doing a previous rsync run), I am seeing a lot of writes on the destination disk, which is something I was not expecting. Does rsync always work like this?

The way I am using it, just in case:

rsync --delete -r /home/my-home /mount-point-other-disk/
3
  • I have no idea how rsync works, but there are at least three better ways of checking for differences between two files than comparing every byte: (1) Is the size different? (2) Is the modification time different? (3) Is the checksum different? It may be better to checksum the files in chunks and break on first difference for a chunk. Also, even checking byte for byte can optimise by breaking after the first mismatch. It may also be threading the comparisons, queuing actions, and updating separately. Commented Dec 12, 2023 at 12:21
  • sure... that sounds like the kind of stuff that I expect rsync to do.
    – eftshift0
    Commented Dec 12, 2023 at 12:31
  • 1
    @Paul_Pedant read my answer for an explanation Commented Dec 12, 2023 at 12:43

1 Answer 1

1

You are copying from what rsync sees as a local disk to a local disk. This switches off many of its optimisations and reduces it almost to a simple cp.

Furthermore, by not including the --times (-t) flag you have switched off its remaining optimisation, meaning that it becomes exactly like cp.

My understanding of rsync (and it might be completely wrong) is that it would first try to read the files (source and destination if it exists) to compare them before attempting to write something on the destination (which I would expect to be slower than reading).

This is only true when you are using rsync to copy across a network between two systems.

I am seeing a lot of writes on the destination disk

This is because rsync cannot assume anything about the destination files and so overwrites them all on every backup attempt.

Explanation:

In the ideal situation, rsync can run itself as a client and also start a server process on the remote system. Assuming the target file appears to be already present on the destination, these two processes each read their local copy and identify by means of blocks of checksums which parts of the files differ. Those differences are then transferred across the network and the destination file updated. (It's a sliding checksum, so you can insert even a single byte at the beginning of the file and that's the only block that will be transferred.)

However, in a situation where rsync is managing reads and writes itself on a single system, it assumes that reading and writing will take the same time. (This is not necessarily true for flash memory, but bear with it.) Consequently, the time taken to read both source and destination file just to compare them - before any writes to update the destination are considered - is broadly equivalent to the time it would take to have just rewritten the destination file, so that's what it does.

Possible solutions:

  1. If you are writing to a NAS, do not write to a NAS filesystem that has been locally mounted. Instead, either use the rsync protocol to write directly across the network to the NAS or else something like ssh to log in to the NAS remotely and transport the data across that connection. In such a case you would want a command line a little like this:

    rsync -a --delete -M--fake-super /home/my-home remoteNAS:/path/to/mount-point-other-disk/
    
  2. If you really are reading/writing between two locally connected disks, include the --times (-t) option and allow rsync to ignore files that look like they have already been copied:

    rsync -a --delete /home/my-home /mount-point-other-disk/
    rsync -rt --delete /home/my-home /mount-point-other-disk/
    

    The first of these copies all metadata, so it not suitable for writing to a non-Linux native filesystem such as NTFS or FAT. The second copies the files but only maintains the time of last modification (no permissions or ownerships). If you are writing to a FAT filesystem note that it can only capture timestamps with a two second accuracy so you need to warn rsync not to expect too much:

    rsync -rt --modify-window 1 --delete /home/my-home /mount-point-other-disk/
    
4
  • Sounds like we have the culprit then. Thanks!
    – eftshift0
    Commented Dec 12, 2023 at 12:48
  • @eftshift0 absolutely. I've added some detail about the "why" for you Commented Dec 12, 2023 at 12:49
  • I have also discovered that using -c can force rsync to do a read/read/compare (compare checksums, anyway) which is what I was expecting to see when I opened the question.... but anyway -t does wonders cause it avoids a lot of the reading in the first place. Thanks!
    – eftshift0
    Commented Dec 13, 2023 at 21:25
  • @eftshift0 yes, that's what forcing the checksum does - but there's no point doing that here: either you trust that the file's correct or you simply rewrite it (in full) Commented Dec 13, 2023 at 22:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .