rsync - What is the difference between using * and ** in the exclude directive?

Question

Following on from my other question about rsync.

I am trying to understand how include/exclude directives use patterns to match against the names of files in a source directory to decide which are to be synced to a destination directory.

Apparently * does not match a directory separator, but ** can match one (zero or more of them, to be precise). Can someone explain very clearly what that means with a few rsync commands?

For example:

Here is my source dir structure:

../openwrt
../openwrt/afile.txt
../openwrt/BackupOfSettings
../openwrt/BackupOfSettings/file1.txt
../openwrt/BackupOfSettings/file2.txt
../openwrt/BackupOfSettings/file3.tar.gz
../openwrt/BackupOfPackages
../openwrt/BackupOfImages
../openwrt/BackupOfImages/anotherfile.txt
../openwrt/BackupOfImages/yetanotherfile.jpg

My source and destination directories:

openWrtPath="/mnt/usb/openwrt/"  
ncpPath="/media/myCloudDrive/openwrt"

Example command:
This command will only sync files with extension: tar.gz from the 'BackupOfSettings' folder.

rsync -vvritn --include='BackupOfSettings/' --include='BackupOfSettings/*.tar.gz' --exclude='*' $openWrtPath $ncpPath

The above command recurses through each sub-dir of the source-dir and applies the include and exclude patterns on each file... so how would it ever "see" a directory separator?

Can someone give a scenario, maybe like the above, that demonstrates the * failing and the ** succeeding in matching a directory separator?

Cheers.

Answer 1

Let's add a few directories and files to BackupOfSettings to test the setup:

mkdir -p "$openWrtPath"/BackupOfSettings/sub1/sub2
touch "$openWrtPath"/BackupOfSettings/file1.tar.gz
touch "$openWrtPath"/BackupOfSettings/sub1/file2.tar.gz
touch "$openWrtPath"/BackupOfSettings/sub1/sub2/file3.tar.gz

And we also add subdirectories sub1 and sub1/sub2 as includes to the command:

rsync -vvritn \
  --include='BackupOfSettings/' \
  --include='BackupOfSettings/sub1/' \
  --include='BackupOfSettings/sub1/sub2/' \
  --include='BackupOfSettings/*.tar.gz' \
  --exclude='*' "$openWrtPath" "$ncpPath"

This syncs BackupOfSettings/file1.tar.gz, BackupOfSettings/sub1/ and BackupOfSettings/sub1/sub2/, but not any files inside the subdirectories as * doesn't match a /.

---

Now let's try BackupOfSettings/**.tar.gz as include:

rsync -vvritn \
  --include='BackupOfSettings/' \
  --include='BackupOfSettings/sub1/' \
  --include='BackupOfSettings/sub1/sub2/' \
  --include='BackupOfSettings/**.tar.gz' \
  --exclude='*' "$openWrtPath" "$ncpPath"

This includes all three *.tar.gz files. The ** is similar to *, but also matches the directory separator / (of sub1/ and sub1/sub2/).

Showing a ** as exclude (as in the title of the question) is a bit difficult, because --exclude='*' excludes the first level below "$openWrtPath" which also means any subdirectories and files are already excluded (because the parent directory at level 1 is excluded). Using ** wouldn't make a difference.

---

To include directory BackupOfSettings and any subdirectories below that directory, you could replace the above three directory includes with

--include='BackupOfSettings/' \
--include='BackupOfSettings/**/' \

or

--include='BackupOfSettings/***/' \

From man rsync:

[...]
o      a ’*’ matches any path component, but it stops at slashes.

o      use ’**’ to match anything, including slashes.

[...]

o      if the pattern contains a / (not counting a trailing /) or a "**", then it
       is  matched  against the full pathname, including any leading directories.
       If the pattern doesn’t contain a / or a "**",  then  it  is  matched  only
       against the final component of the filename.  (Remember that the algorithm
       is applied recursively so "full filename" can actually be any portion of a
       path from the starting directory on down.)

o      a trailing "dir_name/***" will match both the directory (as if "dir_name/"
       had been specified) and everything in the directory (as  if  "dir_name/**"
       had been specified).  This behavior was added in version 2.6.7.
[...]

Answer 2

The double asterisk can be interpreted by the shell as a special globbing character. See also man bash:

globstar
    If set, the pattern ** used in a pathname expansion context will match 
    all files and zero or more directories and subdirectories.  
    If the pattern is followed by  a  /, only directories and subdirectories match.