2

From man rsync:

--existing

skip creating new files on receiver (emphasis mine)

So, what I've done with apparent success:

rsync --existing -rtvhP "remote.machine:/photos/*" /photos

This should reset any changed photos on my local machine to match the originals on the remote.machine (using -rt will act recursively through folders and reset the timestamps, but not bother with chowning or chmodding or copying links--which is what -a does) but NOT transfer any other files--not unchanged ones, and definitely not all the other folders and files from decades of shooting!

Just to be certain of any potential changes that I may have made that changed the data, but didn't modify the filesize/modificationtimestamps (which rsync defaults to use to determine if a file/folder needs to be transferred) I thought I'd add -c (checksum).

rsync --existing -crtvhP "remote.machine:/photos/*" /photos

What I'm noticing is that even if the file/folder is not even present on my local /photos, rsync spends an inordinate amount of time before returning "I didn't do anything". I think this is a bug in rsync where it's not using the results returned from --existing (i.e. "the file does not exists, so therefore do nothing") before progressing on to doing the work it would do if the file DID exist--in this case, checksumming thousands of files that will never need to be transferred over.

FYI, I use photos/* instead of photos/ because I only want the visible files transferred, not any hidden dot.folders/dot.files.

I don't know if it matters, but I'm running rsync in zsh on a Mac locally, talking to a different unix based server; but I verified this condition also exists between homogenous ubuntu systems.

If this is, indeed, a bug in rsync. Can anyone tell me where to best report it?

Thanks for any pointers, advice, answers.

1
  • 1
    Can you confirm this by syncing to an empty directory with --existing and stopping the time with and without -c? This would exclude checksumming of existing files in your destination directory. Building of file lists is not affected by --existing and might take some time for lots of files.
    – Freddy
    Jan 31, 2023 at 20:02

2 Answers 2

2

For this example I'm using: rsync version 2.6.9 protocol version 29

Thanks to @Freddy for the suggestion. It looks like the -c flag takes precedence over --existing, so rsync does a lot of unnecessary (IMHO) work. Here is my test case which, to a newly created empty directory, needs to copy nothing and, I would have thought, not bother checksumming a bunch of stuff it doesn't need to, but it looks like rsync has other preferences (i.e. checksum everything, then decide whether or not it needs to be used.) ¯\(ツ)

I created a new, empty, folder "test", and used the shell built-in time to keep track of how long rsync worked on each test-case of syncing 13104 photos against zero in the new folder.

% mkdir test
% time rsync -rtvhP --existing photos/* test
building file list ...
13104 files to consider
sent 710.56K bytes  received 20 bytes  1.42Mbytes/sec
total size is 912.33G  speedup is 1283921.48
rsync -rtvhP --existing photos/* test  0.03s user 0.09s system 37% cpu 0.327 total

% time rsync -rtvhPc --existing photos/* test
building file list ...
13104 files to consider
sent 919.66K bytes  received 20 bytes  134.90 bytes/sec
total size is 912.33G  speedup is 992002.92
rsync -rtvhPc --existing photos/* test      1201.16s user 210.36s system 20% cpu 1:53:37.01 total

So, it took 0.327 seconds without checksumming, but almost two hours to checksum files it then didn't need to copy. This isn't how I'd hope rsync would work, but it's good to know that it IS how it works. :-)

1
0

Following on from your own answer that confirms --checksum is prioritised over --existing, you can implement a more aggressive version of --existing by providing rsync with a list of files to consider:

Original:

rsync --existing -crtvhP "remote.machine:/photos/*" /photos

Modified variant:

cd /photos &&
find . -type f -print | rsync --dry-run --files-from=- -crtvh --progress 'remote.machine:/photos/' .

You're on a Mac so I've used non-GNU versions of find and rsync. (Of preference I would have used find … -print0 | rsync --from0 … to handle unexpected file names.)

As you would expect, remove --dry-run when you're sure that you've got a working command.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .