1

My question is related to why rsync does not do delta transfer question.

First I create a 1GB file and transfer into /target folder using rsync. Later I added s doo string to the tail of that file. When I transfer the same file into /target folder, I observe that complete file has been transferred instead of transferring only its updated section.

[Q] How could I force rsync to do delta transfer only the updated section of the file? Is is possible or my only option is to transfer complete file all over again?

Example:

$ mkdir target
$ fallocate -l 1G target/temp_1GB_file
$ rsync --inplace --no-whole-file  --size-only --progress temp_1GB_file doo
temp_10GB_file
  1073741824 100%  227.30MB/s    0:00:04 (xfer#1, to-check=0/1)

sent 131163 bytes  received 229425 bytes  48078.40 bytes/sec
total size is 1073741824  speedup is 2977.75

$ echo 'doo' >> temp_10GB_file
$ rsync --no-whole-file  --size-only --progress temp_1GB_file doo # Here complete file has been transferred all over again.
  temp_10GB_file
  1073741828 100%  226.44MB/s    0:00:04 (xfer#1, to-check=0/1)

sent 131171 bytes  received 229418 bytes  48078.53 bytes/sec
total size is 1073741828  speedup is 2977.74
3

You didn't transfer the whole file:

sent 131171 bytes  received 229418 bytes  48078.53 bytes/sec
total size is 1073741828  speedup is 2977.74

The verbose output shows that only a part of the file was ever transferred in the second call to rsync.

rsync starts a sub-process for the sender and one for the receiver, and these sends checksums and data back and forth between themselves.

rsync checksums blocks of the file. By default, the block size depends on the file size. If a checksum for a block mismatches between sender and receiver, the whole block is transferred. The last block was modified by adding a string, so that had to be transferred. This is why you see 131171 bytes being transferred rather than just the number of bytes actually changed in the file (the block size is around 130 KB for this particular file).

See also the -B option in the rsync manual.

  • But if I transfer exactly same file again, verbose output is sent 36 bytes received 20 bytes 112.00 bytes/sec total size is 46080006 speedup is 822857.25 where 36 bytes much less than 131171 bytes. But why adding doo to tail of the file cause 131171 bytes of transfer? @Kusalananda – alper May 15 '18 at 13:34
  • 1
    @Alper rsync does not compare the files byte by byte (which would require sending the whole file), but in blocks of some size. If the checksum for a block has a mismatch, the block is transferred. – Kusalananda May 15 '18 at 13:36
  • Got it. Than probably block size is around 131171 bytes. @Kusalananda – alper May 15 '18 at 13:37
  • 1
    @Alper See the -B option in the rsync manual :-) – Kusalananda May 15 '18 at 13:38
  • @Alper Updated the answer. It's hopefully clearer now. – Kusalananda May 15 '18 at 13:42
1

You didn't transfer the whole file, but both the source and destination were read in their entirety to determine the differences.

With --no-whole-file you traded a simple "replace the file by copying it in its entirety" with a more complex "read the source and calculate its checksums; read the destination and calculate its checksums; update the differences".

On one system I have here it took longer to perform the second operation than just overwriting with the original. (Timings have a variance of about +/- 3 seconds over several attempts.)

  • 29 seconds to copy the original
  • 28 seconds to copy the updated original with the default --whole-file
  • 59 seconds to compare and copy the updated original with --no-whole-file

There are (good) reasons why rsync sometimes does not use the delta transfer algorithm.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.