4

Following on from my other question about rsync.

I am trying to understand how include/exclude directives use patterns to match against the names of files in a source directory to decide which are to be synced to a destination directory.

Apparently * does not match a directory separator, but ** can match one (zero or more of them, to be precise). Can someone explain very clearly what that means with a few rsync commands?

For example:

Here is my source dir structure:

../openwrt
../openwrt/afile.txt
../openwrt/BackupOfSettings
../openwrt/BackupOfSettings/file1.txt
../openwrt/BackupOfSettings/file2.txt
../openwrt/BackupOfSettings/file3.tar.gz
../openwrt/BackupOfPackages
../openwrt/BackupOfImages
../openwrt/BackupOfImages/anotherfile.txt
../openwrt/BackupOfImages/yetanotherfile.jpg

My source and destination directories:

openWrtPath="/mnt/usb/openwrt/"  
ncpPath="/media/myCloudDrive/openwrt"

Example command:
This command will only sync files with extension: tar.gz from the 'BackupOfSettings' folder.

rsync -vvritn --include='BackupOfSettings/' --include='BackupOfSettings/*.tar.gz' --exclude='*' $openWrtPath $ncpPath

The above command recurses through each sub-dir of the source-dir and applies the include and exclude patterns on each file... so how would it ever "see" a directory separator?

Can someone give a scenario, maybe like the above, that demonstrates the * failing and the ** succeeding in matching a directory separator?

Cheers.

2 Answers 2

3

Let's add a few directories and files to BackupOfSettings to test the setup:

mkdir -p "$openWrtPath"/BackupOfSettings/sub1/sub2
touch "$openWrtPath"/BackupOfSettings/file1.tar.gz
touch "$openWrtPath"/BackupOfSettings/sub1/file2.tar.gz
touch "$openWrtPath"/BackupOfSettings/sub1/sub2/file3.tar.gz

And we also add subdirectories sub1 and sub1/sub2 as includes to the command:

rsync -vvritn \
  --include='BackupOfSettings/' \
  --include='BackupOfSettings/sub1/' \
  --include='BackupOfSettings/sub1/sub2/' \
  --include='BackupOfSettings/*.tar.gz' \
  --exclude='*' "$openWrtPath" "$ncpPath"

This syncs BackupOfSettings/file1.tar.gz, BackupOfSettings/sub1/ and BackupOfSettings/sub1/sub2/, but not any files inside the subdirectories as * doesn't match a /.


Now let's try BackupOfSettings/**.tar.gz as include:

rsync -vvritn \
  --include='BackupOfSettings/' \
  --include='BackupOfSettings/sub1/' \
  --include='BackupOfSettings/sub1/sub2/' \
  --include='BackupOfSettings/**.tar.gz' \
  --exclude='*' "$openWrtPath" "$ncpPath"

This includes all three *.tar.gz files. The ** is similar to *, but also matches the directory separator / (of sub1/ and sub1/sub2/).

Showing a ** as exclude (as in the title of the question) is a bit difficult, because --exclude='*' excludes the first level below "$openWrtPath" which also means any subdirectories and files are already excluded (because the parent directory at level 1 is excluded). Using ** wouldn't make a difference.


To include directory BackupOfSettings and any subdirectories below that directory, you could replace the above three directory includes with

--include='BackupOfSettings/' \
--include='BackupOfSettings/**/' \

or

--include='BackupOfSettings/***/' \

From man rsync:

[...]
o      a ’*’ matches any path component, but it stops at slashes.

o      use ’**’ to match anything, including slashes.

[...]

o      if the pattern contains a / (not counting a trailing /) or a "**", then it
       is  matched  against the full pathname, including any leading directories.
       If the pattern doesn’t contain a / or a "**",  then  it  is  matched  only
       against the final component of the filename.  (Remember that the algorithm
       is applied recursively so "full filename" can actually be any portion of a
       path from the starting directory on down.)

o      a trailing "dir_name/***" will match both the directory (as if "dir_name/"
       had been specified) and everything in the directory (as  if  "dir_name/**"
       had been specified).  This behavior was added in version 2.6.7.
[...]
1
  • Thanks Freddy. I get it now. I was focusing on the idea of something like --exclude='**' but I guess it wouldn't make sense to use it that way. Instead the power of **is when used in a directive designed to include or exclude something like an entire branch of a directory tree. Jan 10, 2021 at 0:45
-1

The double asterisk can be interpreted by the shell as a special globbing character. See also man bash:

globstar
    If set, the pattern ** used in a pathname expansion context will match 
    all files and zero or more directories and subdirectories.  
    If the pattern is followed by  a  /, only directories and subdirectories match.
3
  • The question is specifically about its use in rsync filter patterns Jan 9, 2021 at 14:39
  • Ah, sorry, my bad!
    – n0542344
    Jan 9, 2021 at 14:59
  • anyway i think the explanation is same
    – alecxs
    Jan 9, 2021 at 18:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .