-1

I'm testing a script to do a by-directional sync of two directories intelligently using rsync.

Since I'm testing many of the rsync options are not applicable to the testing environment including the lines:

personal_excludes="
--exclude='home/jesse/scripts/'
--exclude='home/jesse/.*/'"

So when I thought I should have -prune on the end of that exclude I really didn't think it would make any difference in my testing environment.

However, adding -prune to these two lines causes rsync to do a DRY-RUN!!?? What The ??

Anyone care to enlighten me on how/why this is? Should I file a bug report for rsync?

Working Directory

jesse@Limbo ~/dev/sync-script-testing $ ls -la . *
-rwxr-x--- 1 jesse jesse 1910 Dec 29 14:04 test-script.sh

.:
total 24
drwxr-x--- 5 jesse jesse 4096 Dec 29 14:05 .
drwxrwxr-x 7 jesse jesse 4096 Dec 29 13:53 ..
drwxr-x--- 2 jesse jesse 4096 Dec 29 13:43 dir1
drwxr-x--- 2 jesse jesse 4096 Dec 29 13:37 dir2
drwxr-x--- 8 jesse jesse 4096 Dec 29 14:03 .git
-rwxr-x--- 1 jesse jesse 1910 Dec 29 14:04 test-script.sh

dir1:
total 12
drwxr-x--- 2 jesse jesse 4096 Dec 29 13:43 .
drwxr-x--- 5 jesse jesse 4096 Dec 29 14:05 ..
-rw-r----- 1 jesse jesse    0 Dec 29 13:33 file2
-rw-r----- 1 jesse jesse    0 Dec 29 13:34 file3
-rw-r----- 1 jesse jesse    0 Dec 29 13:35 file4
-rw-r----- 1 jesse jesse   32 Dec 29 14:04 .lastsync

dir2:
total 12
drwxr-x--- 2 jesse jesse 4096 Dec 29 13:37 .
drwxr-x--- 5 jesse jesse 4096 Dec 29 14:05 ..
-rw-r----- 1 jesse jesse    0 Dec 29 13:33 file1
-rw-r----- 1 jesse jesse    0 Dec 29 13:34 file3
-rw-r----- 1 jesse jesse    0 Dec 29 13:35 file4
-rw-r----- 1 jesse jesse   32 Dec 29 14:04 .lastsync

Script Contents

jesse@Limbo ~/dev/sync-script-testing $ cat test-script.sh 
#!/usr/bin/bash

#rsync -av --files-from=<(cd dir1 && find ./ -newermt "$(sed 's/^Successful sync on //' sync.log)") --exclude=/sync.log ./dir1/ ./dir2/ && echo "Successful sync on $(date -R)" | tee dir2/sync.log > dir1/sync.log
#
#        # Perform the Sync
#        echo -e "\n[1] Uppdate HDD -> USB   DO NOT Include configuration .dot file - They take too long\n"

# First do a sync of ONLY those files modified since last run WITHOUT deleting any files

rsync_general_args="
 --verbose
 --human-readable
 --progress
 --recursive
 --update
 --links
 --perms
 --times
 --group
 --owner
 --devices
 --specials
 --hard-links
 --xattrs
 --one-file-system
 --one-file-system" # specifying --one-file-system twice means something different!

personal_excludes="
 --exclude='home/jesse/scripts/'
 --exclude='home/jesse/.*/'"

# Build list of files to sync
# First use mktemp to create a randomly named temporary directory to keep tempfiles in. This prevents hackers from intercepting data
tmpdir=$(mktemp -d "${TMPDIR:-/tmp/}$(basename $0).XXXXXXXXXXXX")

# Find files modified since the last sync, date saved in .lastcync
# Do this in a subshell so as not to change the current directory
$(cd dir1 && find ./ -newermt "$(cat .lastsync)" ! -path "./.lastsync" >$tmpdir/rsync_files_to_sync_from_dir1)
$(cd dir2 && find ./ -newermt "$(cat .lastsync)" ! -path "./.lastsync" >$tmpdir/rsync_files_to_sync_from_dir2)

echo; echo --files-from dir1=$(cat $tmpdir/rsync_files_to_sync_from_dir1)
rsync $rsync_general_args $personal_excludes --files-from=$tmpdir/rsync_files_to_sync_from_dir1 --exclude=/.lastsync ./dir1/ ./dir2/ && date -R > ./dir1/.lastsync

echo; echo --files-from dir2=$(cat $tmpdir/rsync_files_to_sync_from_dir2)
rsync $rsync_general_args $personal_excludes --files-from=$tmpdir/rsync_files_to_sync_from_dir2 --exclude=/.lastsync ./dir2/ ./dir1/ && date -R > ./dir2/.lastsync


# Clean Up
rm -r $tmpdir

Script output

jesse@Limbo ~/dev/sync-script-testing $ ./test-script.sh 

--files-from dir1=
sending incremental file list

sent 18 bytes  received 12 bytes  60.00 bytes/sec
total size is 0  speedup is 0.00

--files-from dir2=
sending incremental file list

sent 18 bytes  received 12 bytes  60.00 bytes/sec
total size is 0  speedup is 0.00

Make a change to the script: Adding "-prune"

jesse@Limbo ~/dev/sync-script-testing $ git log -p --grep WHY??
commit d9d50bdd289616faaf3d174e6023ba16a5286b53
Author: Jesse the Wind Wanderer <webmaster@windwanderer.com.au>
Date:   Sat Dec 29 14:03:43 2018 +0800

    adding '-prune' causes rsync to do DRY-RUN. WHY???

diff --git a/test-script.sh b/test-script.sh
index 37ee2e6..b45f4a0 100755
--- a/test-script.sh
+++ b/test-script.sh
@@ -26,8 +26,8 @@ rsync_general_args="
  --one-file-system" # specifying --one-file-system twice means something different!

 personal_excludes="
- --exclude='home/jesse/scripts/'
- --exclude='home/jesse/.*/'"
+ --exclude='home/jesse/scripts/' -prune
+ --exclude='home/jesse/.*/' -prune"

 # Build list of files to sync
 # First use mktemp to create a randomly named temporary directory to keep tempfiles in. This prevents hackers from intercepting data

Script now runs rsync with DRY-RUN

jesse@Limbo ~/dev/sync-script-testing $ ./test-script.sh 

--files-from dir1=
sending incremental file list
./
file2

sent 123 bytes  received 22 bytes  290.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)

--files-from dir2=
sending incremental file list
./
file1

sent 127 bytes  received 22 bytes  298.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)
  • The $(cd dir1 && find ./ -newermt "$(cat .lastsync)" ! -path "./.lastsync" >$tmpdir/rsync_files_to_sync_from_dir1) syntax works by chance. It says run cd dir1 && find ./ -newermt "$(cat .lastsync)" ! -path "./.lastsync" >$tmpdir/rsync_files_to_sync_from_dir1 in a subshell and execute the output. Because the output of the find is sent to a file, and you don't have things like CDPATH set the output of the subshell is empty. You should remove the $( and ) – icarus Dec 29 '18 at 8:34
  • on an unrelated note, are you sure personal_excludes="--exclude='home/jesse/scripts/'"; rsync $personal_excludes works? You're passing literal single-quotes to rsync (and the strings are theoretically subject to globbing). This looks like one of those cases where you'd be better off using arrays. – ilkkachu Dec 29 '18 at 11:45
2

Ohhhhhhhhh, Now I feel silly. So as a lesson to myself and others I'll post this. -prune is an option of find not rsync!! find is unusual in that it only uses single '-' with long-options.

Adding -prune to rsync meant that each of the individual letters were interpreted as a single option '-p' '-r' '-u' '-n' (which = Dry-Run!!) and '-e'

  • Funny that all those options existed, otherwise you would have gotten a proper error message, like for Q: rsync: -Q: unknown option :-) – nst0022 Dec 29 '18 at 7:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.