0

Here is my source dir structure:

../openwrt
../openwrt/BackupOfPackages
../openwrt/BackupOfSettings
../openwrt/BackupOfImages

I want to use rsync to mirror that to the destination dir. I only want to rsync files in the three sub-directories and I only want to sync files with extension:
.installed from the BackupOfPackages folder
.img.gz from the BackupOfImages folder
.tar.gz from the BackupOfSettings folder

In my source dir I have a mixture of all sorts of files in the basedir "openwrt" and in each of the sub dirs. However as above, I only want to sync specific file types from specific sub-dirs.

enter image description here

Here is my rsync command:

openWrtPath="/mnt/usb/openwrt/"  
ncpPath="/media/myCloudDrive/openwrt"

rsync -vvrit --include='/' --include='BackupOfSettings/' --include='BackupOfSettings/**.tar.gz' --include='BackupOfPackages/' --include='BackupOfPackages/**.installed' --include='BackupOfImages/' --include='BackupOfImages/**.img.gz' --exclude='/*' --delete --delete-excluded $openWrtPath ncp:$ncpPath

When I run the command this is what it shows and hides. It correctly hides opkg.installed and "New Folder" from the basedir but does not hide file types such as .installed from BackupOfSettings and BackupOfImages etc..

enter image description here

Rsync syncs almost everything from the source to the destination? enter image description here

How do I change my rsync command to only copy files with extension:
.installed to the BackupOfPackages folder
.img.gz to the BackupOfImages folder
.tar.gz to the BackupOfSettings folder
i.e: To correctly hide any file types that should not be transferred to the destination from the various sub-dirs even if they exist in those sub dirs on the source machine.

UPDATE

Right after I posted I think I found the answer.. when I change --exclude='/*' to --exclude='*' it seems to work as I want. Can someone explain the difference between those two excludes?

Does '/*' mean: "Exclude everything in the basedir (openwrt) and this does not apply to sub-directories of the basedir"?

Thank you!

3
  • 1
    Please don't post pictures of text. It's really hard to see what's going on
    – roaima
    Jan 7 at 8:14
  • I take your point and read the link. But I am also pretty sure you would have a perfectly clear view of the screenshot if you had simply clicked on each one. Jan 7 at 20:40
  • 1
    You have no idea what my vision may or may not be like
    – roaima
    Jan 7 at 21:03
1

Does '/*' mean: "Exclude everything in the basedir (openwrt) and this does not apply to sub-directories of the basedir"?

Yes, / is the root of the transfer directory and /* matches everything there (files and directories).

You can shorten the includes in your command to:

rsync -ai \
  --include='BackupOf***/' \
  --include='BackupOfImages/**.img.gz' \
  --include='BackupOfSettings/**.tar.gz' \
  --include='BackupOfPackages/**.installed' \
  --exclude='*' \
  --delete --delete-excluded \
  "$openWrtPath" ncp:"$ncpPath"

Here --include='BackupOf***/' means to include all directories starting with BackupOf and its subdirectories (e.g. BackupOfImages/sub1/sub2/). If you don't need to include those subdirectories, change it to BackupOf*/ and you can also change the two **'s of the file extensions to one *.

I used the archive mode -a which includes -t and other options and quoted both variables.

1
  • Cheers. I had to run a few variations of that rsync command to understand the difference but I think I finally got it! Basically --exclude='/*'excludes all files and folders that have not been explicitly included. But the exclusion is not extended to the contents of the folders. So in my command above I still see ALL files in the subfolders. Whereas --exclude='*'`excludes files and folders and sub folders and the contents of those sub folders etc.. Jan 7 at 21:31
0

rsync doesn't include a / at the start of the paths it considers. Why? designer's choice.

But there is another issue. You've used ** in your includes to include files in subdirectories. However, the subdirectories themselves are excluded. Try changing the includes and excludes to:

--include='*/' --include='BackupOfSettings/**.tar.gz' --include='BackupOfPackages/**.installed' --include='BackupOfImages/**.img.gz' --exclude='*' --prune-empty-dirs

This is: include all directories, only the files you want, no other files, and drop the directories that are empty.

The other option would be to write:

--include='BackupOfSettings/' --include='BackupOfSettings/**/' --include='BackupOfSettings/**.tar.gz' --include='BackupOfPackages/' --include='BackupOfPackages/**/' --include='BackupOfPackages/**.installed' --include='BackupOfImages/' --include='BackupOfImages/**/' --include='BackupOfImages/**.img.gz' --exclude='*'

In this case, this is including the specific directories, subdirectories thereof, and the wanted files. This variant would be needed if you have a large number of other directories in the source tree that you wish to avoid searching.

2
  • 1
    @FlexMcMurphy The point of the firrst sentence is: The designers of rsync chose not to include a leading slash in the path being matched. It is relative to the source path specified, and does not include a leading slash. Thus, if you have file /a/b/c/d and specify rsync /a/b/ dest/ then the path transferred will be c/d and not /c/d. This was a design decision. The point of including it was that you asked why "*" instead of "/*".
    – David G.
    Jan 8 at 2:14
  • Thank you for clarifying this. Jan 8 at 21:34
0

To expand on the accepted answer by @Freddy

My source and destination directories:

openWrtPath="/mnt/usb/openwrt/"  
ncpPath="/media/myCloudDrive/openwrt"

My goal is to only sync files with extension .tar.gz from a subfolder in the source directory called: BackupOfSettings. Subfolders in the BackupOfSettings subfolder are also ignored.

Here is a simplified version of the command in the question. Credit here and here for explanation of how rsync will sync a specific file type from a specific sub-dir.

rsync -vvritn --include='BackupOfSettings/' --include='BackupOfSettings/*.tar.gz' --exclude='*' $openWrtPath ncp:$ncpPath
  1. To achieve this with rsync you need to first include the sub-directory that contains the files you want: --include='BackupOfSettings/'
  2. Then in a separate include you need to define a pattern in that same sub-directory that will match your desired file type: --include='BackupOfSettings/*.tar.gz'
  3. Then you need to exclude everything else using: --exclude='*' which matches and excludes files and folders that are not explicitly included.

--exclude='/*' vs --exclude='*'

rsync -vvritn --include='BackupOfSettings/' --include='BackupOfSettings/*.tar.gz' --exclude='/*' $openWrtPath ncp:$ncpPath

If we use --exclude='/*' then the above command will allow all files in the BackupOfSettings sub-folder because this exclude only matches on files and sub-folders in the source directory... it does not match files in sub-folders of the source directory.

6
  • Your understanding is flawed. * is a wildcard pattern matching anything except the directory separator /. Similarly a*sh will match anything (except /) starting with the letter a and ending with sh, regardless of file or directory ,(or special). A leading / anchors the wildcard pattern to the top of the source path. A trailing /` anchors to any directory. In your last sentence, "[--exclude='/*'] does not match anything in sub-folders of the source directory" Finally, it's really important to realise that the include and exclude directives are applied in order
    – roaima
    Jan 8 at 8:30
  • * is a wildcard pattern matching anything except the directory separator /- If my source folder contains a subfolder called testfolder, that is not explicitly included prior to --exclude='*', the verbose output of rsync shows that testfolder is hidden because of pattern *. Doesn't this mean that the wildcard pattern does match a directory separator? Jan 8 at 21:33
  • No. The wildcard has matched the directory name testfolder. That pattern doesn't contain a directory separator
    – roaima
    Jan 8 at 22:49
  • I don't understand why it has to be clarified that the wildcard pattern *does not match a directory separator. In my command above rsync recurses into the BackupOfSettings directory and the include and exclude directives are applied to all files in that dir right? So how would it ever see a directory separator? A filename cannot include a slash can it? Can you give an example? Jan 9 at 12:45
  • 1
    I can't conceptualize the * failing or **succeeding in matching a directory separator. I asked another question about that and hopefully someone could give a few rsync commands to illustrate it. Cheers. Jan 9 at 14:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.