0

I'm trying to understand the output of rsync -n --update -av dir1/ dir2/. I've been told that this should list all directories, plus files which either don't exist in dir2 or are newer in dir1. I've tried this on a test directory:

test
test/testDir1
test/testDir1/testDir11
test/testDir1/testDir11/testFile3.txt
test/testDir1/testFile2.txt
test/testFile1.txt
test/testDir2

If I copy this to testCopy, change test/testDir1/testFile2.txt, then run rsync -n --update -av test/ testCopy/, the output is:

./
testDir1/
testDir1/testFile2.txt
testDir1/testDir11/
testDir2/

Which is what I would expect based on what I was told about how the output should behave.

However, I've tried this same command on other much larger pairs of real directories, getting much different behavior. Take two examples of pairs of directories which should be synced:

  • In one case, the output is merely ./, despite the fact that there are many directories (620) and files. No files are listed.

  • For another, some (7/72) directories are listed. No files are listed.

What am I not understanding about what this output is showing me?

  • Do you have a source for "this should list all directories"? My understanding is it only lists directories which need to be transferred, ie are new or have changed (e.g. in attributes or timestamp) - same as for regular files. Not all directories. – Joe P Jun 22 '17 at 12:22
  • @JoeP No, I don't have a source at all, just what I was told by a colleague. If you can point me toward some documentation (I haven't tried reading through the entire man page), that would be really helpful. That being said, if it were just listing new/changed directories, wouldn't my first example exclude testDir1/testDir11/ and testDir2/ from the output? – Evan Jun 22 '17 at 12:32
  • Sorry, I haven't found clear documentation on this myself yet! The relevant bits of man rsync are --dry-run (produces the --verbose and/or --itemize-changes output) and --verbose (which doesn't say anything specific about directories). The key to understanding it might be to look at what changes rsync sees in a directory. In particular, if you create a file inside a directory, that directory gets its timestamp updated - so it's now a candidate for synchronising - but its parent directory doesn't. In fact, I think that might be the answer. – Joe P Jun 22 '17 at 12:41
1

Although I don't have definitive documentation to point to, this is what I think is going on.

With -v|--verbose, files that are to be transferred (or would be transferred, when -n|--dry-run was specified) are listed. Directories are just files in Unix-like OSes and they're listed on the same rules.

So, when are directories transferred? (Apart from when they don't exist on the target obviously.) Assuming you specify -t|--times (which is implied by -a), they'll be transferred when their timestamps change. Similar rules if their attributes have changed.

And when do directories change timestamps? By observation this is (at least) when a file in the directory is created or renamed, but not when a file just changes its attributes eg its time.

So if you touch a new file in testDir11, testDir11 will change timestamp and become transferrable. But if you touch an existing file, the directory doesn't change.

  • After some more testing, it looks like directories get listed when any attribute is different between the source and the destination, even when the destination timestamp is newer. For example, if I modify the file testCopy/testDir1/testFile2.txt, then testCopy/testDir1/ gets listed. Which is really strange to me. I think this is what is causing the confusion in my real examples: in the first case, I added /test/ and /testCopy/, so ./ is listed. In the second case, some files were added or changed in 7 of the 72 total directories - which I verified by switching src and dest. – Evan Jun 22 '17 at 14:07
2

Bottom line:

  1. A copy (the command cp) does a different job than rsync. For one, it does not copy the timestamps.

  2. The rsync option --update will update only newer files.


Assuming that what you did was a copy of directories, in specific, this:

#!/bin/bash

mkdir -p test/testDir2
mkdir -p test/testDir1/testDir11
echo "yes3" > test/testDir1/testDir11/testFile3.txt
echo "yes2" > test/testDir1/testFile2.txt
echo "yes1" > test/testFile1.txt

sleep 3
cp -r test/ testcopy/

That will create two directories with a similar structure:

$ tree test
test
├── testDir1
│   ├── testDir11
│   │   └── testFile3.txt
│   └── testFile2.txt
├── testDir2
└── testFile1.txt

And, the copy:

$ tree testCopy/
testCopy/
├── testDir1
│   ├── testDir11
│   │   └── testFile3.txt
│   └── testFile2.txt
├── testDir2
└── testFile1.txt

The directories look equal, but are not. The sleep 3 before the cp will make all files in testCopy with different times:

$ ls -la --time-style=full-iso test/testFile1.txt
-rw-r--r-- 1 user user 5 2017-06-22 20:22:13.704512555 0000 test/testFile1.txt

$ ls -la --time-style=full-iso testCopy/testFile1.txt
-rw-r--r-- 1 user user 5 2017-06-22 20:22:16.716355225 0000 testCopy/testFile1.txt

Then, testing rsync with the options you stated (nothing will be changed) will show this:

$ rsync -n --update -va test/ testCopy/
sending incremental file list
./
testDir1/
testDir1/testDir11/
testDir2/

Why is that so?

Because the --update option takes precedence and will only sync new files. The files in the copy are newer than the ones in the original.

If we touch a file in the original directory, it will be included:

$ touch test/testFile1.txt
$ rsync -n --update -va test/ testCopy/
sending incremental file list
./
testFile1.txt
testDir1/
testDir1/testDir11/
testDir2/

What will make a full sync (older or newer but different) will be:

$ rsync -n -va test/ testCopy/
sending incremental file list
./
testFile1.txt
testDir1/
testDir1/testFile2.txt
testDir1/testDir11/
testDir1/testDir11/testFile3.txt
testDir2/

If that is allowed to execute the sync (remove the -n option), and run again:

$ rsync -n -va test/ testCopy/
sending incremental file list

Nothing needs to be exchanged as everything got updated.

If one file is changed (make it newer), it will be updated:

$ touch test/testDir1/testFile2.txt
$ rsync -n --update -va test/ testCopy/
sending incremental file list
testDir1/testFile2.txt

But after an actual sync, if the file made newer is the one in the copy. it will not need to be exchanged ( it is already newer).

$ touch testCopy/testDir1/testFile2.txt
$ rsync -n --update -va test/ testCopy/
sending incremental file list
  • Couple questions: (1) Should there be a sleep 3 in the first code block? (2) How are the files in the copy older than the ones in the original? Did you mean it the other way around? (3) Why are the directories listed in the first rsync, and not the files? (4) If I touch a folder in the copy, then run the rsync, that folder gets listed, even though it is in the destination rather than the source. Why does that folder get listed? – Evan Jun 23 '17 at 19:38
  • @Evan (1) yes, answer updated shortly (2) sorry, meant: the copy is newer (a bigger time) (3) because that is the way rsync works (directories and newer files) and (4) because the folder is different and needs updating to become equal (last information known about the directory). – Arrow Jun 23 '17 at 20:08
  • Cool. For (3) and (4) - are you aware of a reason why rsync should work this way? I would have expected --update to modify the behavior for both files and directories, rather than just files. I expect there is a good reason, although I wouldn't be entirely surprised if it's just a quirk. – Evan Jun 23 '17 at 20:16
  • @Evan It works as intended: the manual says: skip files that are newer on the receiver. So, everything is the same, but only skip files that are newer in the receiver. I do not have the detail of the why, but it seems reasonable to me when come to think about it. – Arrow Jun 23 '17 at 20:23
  • Woaw, not even one up-vote? I should be quite bad at writing answers. – Arrow Jun 23 '17 at 20:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.