1

Following on from my other question about rsync.

I am trying to understand how include/exclude directives use patterns to match against the names of files in a source directory to decide which are to be synced to a destination directory.

Apparently * does not match a directory separator, but ** can match one (zero or more of them, to be precise). Can someone explain very clearly what that means with a few rsync commands?

For example:

Here is my source dir structure:

../openwrt
../openwrt/afile.txt
../openwrt/BackupOfSettings
../openwrt/BackupOfSettings/file1.txt
../openwrt/BackupOfSettings/file2.txt
../openwrt/BackupOfSettings/file3.tar.gz
../openwrt/BackupOfPackages
../openwrt/BackupOfImages
../openwrt/BackupOfImages/anotherfile.txt
../openwrt/BackupOfImages/yetanotherfile.jpg

My source and destination directories:

openWrtPath="/mnt/usb/openwrt/"  
ncpPath="/media/myCloudDrive/openwrt"

Example command:
This command will only sync files with extension: tar.gz from the 'BackupOfSettings' folder.

rsync -vvritn --include='BackupOfSettings/' --include='BackupOfSettings/*.tar.gz' --exclude='*' $openWrtPath $ncpPath

The above command recurses through each sub-dir of the source-dir and applies the include and exclude patterns on each file... so how would it ever "see" a directory separator?

Can someone give a scenario, maybe like the above, that demonstrates the * failing and the ** succeeding in matching a directory separator?

Cheers.

1

Let's add a few directories and files to BackupOfSettings to test the setup:

mkdir -p "$openWrtPath"/BackupOfSettings/sub1/sub2
touch "$openWrtPath"/BackupOfSettings/file1.tar.gz
touch "$openWrtPath"/BackupOfSettings/sub1/file2.tar.gz
touch "$openWrtPath"/BackupOfSettings/sub1/sub2/file3.tar.gz

And we also add subdirectories sub1 and sub1/sub2 as includes to the command:

rsync -vvritn \
  --include='BackupOfSettings/' \
  --include='BackupOfSettings/sub1/' \
  --include='BackupOfSettings/sub1/sub2/' \
  --include='BackupOfSettings/*.tar.gz' \
  --exclude='*' "$openWrtPath" "$ncpPath"

This syncs BackupOfSettings/file1.tar.gz, BackupOfSettings/sub1/ and BackupOfSettings/sub1/sub2/, but not any files inside the subdirectories as * doesn't match a /.


Now let's try BackupOfSettings/**.tar.gz as include:

rsync -vvritn \
  --include='BackupOfSettings/' \
  --include='BackupOfSettings/sub1/' \
  --include='BackupOfSettings/sub1/sub2/' \
  --include='BackupOfSettings/**.tar.gz' \
  --exclude='*' "$openWrtPath" "$ncpPath"

This includes all three *.tar.gz files. The ** is similar to *, but also matches the directory separator / (of sub1/ and sub1/sub2/).

Showing a ** as exclude (as in the title of the question) is a bit difficult, because --exclude='*' excludes the first level below "$openWrtPath" which also means any subdirectories and files are already excluded (because the parent directory at level 1 is excluded). Using ** wouldn't make a difference.


To include directory BackupOfSettings and any subdirectories below that directory, you could replace the above three directory includes with

--include='BackupOfSettings/' \
--include='BackupOfSettings/**/' \

or

--include='BackupOfSettings/***/' \

From man rsync:

[...]
o      a ’*’ matches any path component, but it stops at slashes.

o      use ’**’ to match anything, including slashes.

[...]

o      if the pattern contains a / (not counting a trailing /) or a "**", then it
       is  matched  against the full pathname, including any leading directories.
       If the pattern doesn’t contain a / or a "**",  then  it  is  matched  only
       against the final component of the filename.  (Remember that the algorithm
       is applied recursively so "full filename" can actually be any portion of a
       path from the starting directory on down.)

o      a trailing "dir_name/***" will match both the directory (as if "dir_name/"
       had been specified) and everything in the directory (as  if  "dir_name/**"
       had been specified).  This behavior was added in version 2.6.7.
[...]
1
  • Thanks Freddy. I get it now. I was focusing on the idea of something like --exclude='**' but I guess it wouldn't make sense to use it that way. Instead the power of **is when used in a directive designed to include or exclude something like an entire branch of a directory tree. Jan 10 at 0:45
-1

The double asterisk can be interpreted by the shell as a special globbing character. See also man bash:

globstar
    If set, the pattern ** used in a pathname expansion context will match 
    all files and zero or more directories and subdirectories.  
    If the pattern is followed by  a  /, only directories and subdirectories match.
3
  • The question is specifically about its use in rsync filter patterns
    – roaima
    Jan 9 at 14:39
  • Ah, sorry, my bad!
    – n0542344
    Jan 9 at 14:59
  • anyway i think the explanation is same
    – alecxs
    Jan 9 at 18:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.