7

Can I get rsync to work under the following conditions?

if len(f1) != len(f2) then rsync
if len(f1) == len(f2) and md5sum(f1) != md5sum(f2) then rsync

The closest is the --checksum option??


2
  • 3
    Yes, --checksum (or -c) is exactly what you want. May 24, 2017 at 10:00
  • 2
    It would be worth reviewing whether you can revert to trusting timestamps (for example, use cp -p and scp -p instead of just cp or scp when copying files around). The consequent efficiency improvement within rsync is vast. May 24, 2017 at 11:11

1 Answer 1

9

Taken from rsync manpage:

Description

Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.

Rsync finds files that need to be transferred using a lqquick checkrq algorithm (by default) that looks for files that have changed in size or in last-modified time. Any changes in the other preserved attributes (as requested by options) are made on the destination file directly when the quick check indicates that the file's data does not need to be updated.

So, the default behavior as we can see on the description is:

  • Copy tool, that works locally or remotelly
  • Lots of options
  • Delta-transfer algorithm by default, that will only transfer sets of files that differ to reduces the amount of network usage
  • Widely used tool for mirroring and backups
  • checkrq algorithm that does what you want on condition 1: if len(f1) != len(f2) then rsync
  • Destination is the one to be affected if no options are passed.

Now, it is just a matter of looking for options related to checksum. Searching at the manuals:

-c, --checksum
   This changes the way rsync checks if the files have been changed and are in
   need of a transfer. Without this option, rsync uses a lqquick checkrq that
   (by default) checks if each file's size and time of last modification match
   between the sender and receiver. This option changes this to compare a 128-
   bit checksum for each file that has a matching size. Generating the checksums
   means that both sides will expend a lot of disk I/O reading all the data in
   the files in the transfer (and this is prior to any reading that will be
   done to transfer changed files), so this can slow things down significantly.

The description of --checksum is exactly what you want on if len(f1) == len(f2) and md5sum(f1) != md5sum(f2) then rsync. It will do a 128-bit checksum on each size matching file.

But be careful cause this option will, depending on the situation, increase you I/O significantly.

4
  • 1
    Might be worth adding a note to remind the OP that it adds no value whatsoever if using rsync between locally mounted filesystems. In such situations rsync reverts to cp -p. May 24, 2017 at 11:13
  • Will it? I found no explicit evidence inside rsync documentation that it will revert to cp -p under those circumstances(local file copy). It will still apply locally its delta algorithm, and it will just behave like cp if --perms and --executability options are passed. Source - download.samba.org/pub/rsync/rsync.html
    – user34720
    May 24, 2017 at 18:56
  • If you're doing a local file copy the delta algorithm is explicitly disabled (and cannot be enabled unless you cheat with something like rsync -a /path/to/source localhost:/path/to/dest). May 24, 2017 at 19:08
  • Ok. Found at the --whole-file parameter that explains your point of view. However, you will still benefit from time modification and size comparsions that cp -p will not provide you...
    – user34720
    May 24, 2017 at 19:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .