3

I made a backup of one of my computer's internal drives to an external. The next time I used rsync to sync the two drives, I noticed that large (40 + GB) files that weren't modified were still taking a long time to "copy". I thought rsync looked at mod-times and file size first? Why would it take so long; as though it were using checksum?

I had originally copied the files using rsync -rv --delete /src/path/ /dest/path/

Improve this question
8

Since you are not copying the metadata (which you would do if you used --archive or -a instead of just -r), the metadata (timestamps, ownerships etc.) will be different between the copy and the original. When you run rsync again, since the timestamps are different, the file is copied again.

So, you would instead want to use

rsync -ai --delete /src/path/ /dest/path

I'm using -i (--itemize-changes) since it also tells me why a file was copied.

Also note that when you do a local copy with rsync, it will not use its delta algorithm, but will instead behave as if --whole-file (or -W) was specified. This is because the delta algorithm is assumed to only be faster than a whole file transfer when transferring over a network. When using the delta algorithm, the whole file would need to be read and checksummed on both the source and target systems. Doing this locally seems a bit wasteful, so the file is just copied in full instead.

Improve this answer
  • I just tested locally with a large file containing random bits, and I find that you are correct; the -a flag is necessary. This is actually kind of annoying; why would the main selling point of the tool not be its default behavior? Fortunately I can 'mend' this by recopying with the -a flag. Thanks for your answer – DeepDeadpool Nov 5 '18 at 16:30
  • @DeepDeadpool Because of performance reasons, as I mentioned. You could run it with --no-whole-file or --no-W if you wished to use the delta algorithm for a local transfer. – Kusalananda♦ Nov 5 '18 at 16:33
  • 1
    @DeepDeadpool Note also that if you're copying over a network with -r instead of -a, then, when you invoke rsync again, it would be forced to use the delta algorithm, whereas if you had used -a it could just have skipped that transfer completely. – Kusalananda♦ Nov 5 '18 at 16:36
  • I'm using -a and it's still dog slow... – Joseph Garvin Mar 13 '20 at 22:05
  • @JosephGarvin If it's an initial copy, then all data has to be copied. Any later copy with updated data at the source, against the same destination, will be faster. – Kusalananda♦ Mar 13 '20 at 23:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.