Rsync filter: copying one pattern only

Question

I am trying to create a directory that will house all and only my PDFs compiled from LaTeX. I like keeping each project in a separate folder, all housed in a big folder called LaTeX. So I tried running:

rsync -avn *.pdf ~/LaTeX/ ~/Output/

which should find all the pdfs in ~/LaTeX/ and transfer them to the output folder. This doesn't work. It tells me it's found no matches for "*.pdf". If I leave out this filter, the command lists all the files in all the project folders under LaTeX. So it's a problem with the *.pdf filter. I tried replacing ~/ with the full path to my home directory, but that didn't have an effect.

I'm, using zsh. I tried doing the same thing in bash and even with the filter that listed every single file in every subdirectory... What's going on here?

Why isn't rsync understanding my pdf only filter?

---

OK. So update: No I'm trying

rsync -avn --include="*/" --include="*.pdf" LaTeX/ Output/

And this gives me the whole file list. I guess because everything matches the first pattern...

Answer 1

Rsync's filter rules can seem daunting when you read the manual, but there are a few simple principles that suffice in many cases:

• Inclusions and exclusions:

  • Excluding files by name or by location is easy: --exclude=*~, --exclude=/some/relative/location.
  • If you only want to match a few files or locations, include them, include every directory leading to them (for example with --include=*/), then exclude the rest with --exclude='*'. This is because:
  • If you exclude a directory, this excludes everything below it.
  • If you include a directory, this doesn't automatically include its contents. In recent versions, --include='directory/***' will do that.
  • For each file, the first matching rule applies (and anything never matched is included).

• Patterns:

  • If a pattern doesn't contain a /, it applies to the file name sans directory.
  • If a pattern ends with /, it applies to directories only.
  • If a pattern starts with /, it applies to the whole path from the directory that was passed as an argument to rsync.
  • * any substring of a single directory component (i.e. never matches /); ** matches any path substring.

• If a source argument ends with a /, its contents are copied (rsync -r a/ b creates b/foo for every a/foo). Otherwise the directory itself is copied (rsync -r a b creates b/a).

---

Thus here we need to include *.pdf, include directories containing them, and exclude everything else.

rsync -a --include='*.pdf' --include='*/' --exclude='*' ~/LaTeX/ ~/Output/

Note that this copies all directories, even the ones that contain no matching file or subdirectory containing one. This can be avoided with the --prune-empty-dirs option (it's not a universal solution since you then can't copy a directory even by matching it explicitly, but that's a rare requirement).

rsync -am --include='*.pdf' --include='*/' --exclude='*' ~/LaTeX/ ~/Output/

Answer 2

You can use find and an intermediate list of files (files_to_copy) to solve your issue. Make sure you're in your home directory, then:

find LaTeX/ -type f -a -iname "*.pdf" > files_to_copy && rsync -avn --files-from=files_to_copy ~/ ~/Output/ && rm files_to_copy

Tested with Bash.

Answer 3

If you use a pattern like *.pdf, the shell “expands“ that pattern, i.e. it replaces the pattern with all matches in the current directory. The command you are running (in this case rsync) is unaware of the fact that you tried to use a pattern.

When you are using zsh, there is an easy solution, though: The ** pattern can be used to match folders recursively. Try this:

rsync -avn ~/LaTeX/**/*.pdf ~/Output/

Answer 4

Judging by the "INCLUDE/EXCLUDE PATTERN RULES" section of the manpage, the way to do this is

rsync -avn --include="*/" --include="*.pdf" ~/Latex/ ~/Output/

The critical difference between this and kbrd's answer is the --include="*/" flag, which tells rsync to go ahead and copy any directories it finds, whatever they are named. This is needed because rsync will not recurse into a subdirectory unless it has been instructed to copy that subdirectory.

Also, note that the quotation marks prevent the shell from trying to expand the patterns to filenames relative to the current directory, and doing one of the following:

1. Succeeding and messing up your filter (not too likely in the middle of a flag like that, though you really never know when someone will make a file named --include=foo.pdf ...)

2. Failing, and potentially producing an error instead of running the command (as you've discovered zsh does by default).

Answer 5

How about this:

rsync -avn --include="*.pdf" ~/Latex/ ~/Output/

Answer 6

rsync -av --include="*/" --include="*.pdf" --exclude="*" ~/Latex/ ~/Output/ --dry-run

The default is to include everything, so you must explicitly exclude everything after including the files you want to transfer. Remove the --dry-run to actually transfer the files.

If you start off with:

--exclude '*' --include '*.pdf'

Then the greedy matching will exclude everything right off.

If you try:

--include '*.pdf' --exclude '*' 

Then only pdf files in the top level folder will be transferred. It won't follow any directories, since those are excluded by '*'.

Answer 7

Here is something that should work without using find. The difference from answers already posted is the order of the filter rules. Filter rules in an rsync command work a lot like iptable rules, the first rule that a file matches is the one that is used. From the manual page:

As the list of files/directories to transfer is built, rsync checks each name to be transferred against the list of include/exclude patterns in turn, and the first matching pattern is acted on: if it is an exclude pattern, then that file is skipped; if it is an include pattern then that filename is not skipped; if no matching pattern is found, then the filename is not skipped.

Thus, you need a command as follows:

rsync -avn --include="**.pdf" --exclude="*" ~/LaTeX/ ~/Output/

Note the "**.pdf" pattern. According to the man page:

if the pattern contains a / (not counting a trailing /) or a "**", then it is matched against the full pathname, including any leading directories. If the pattern doesn’t contain a / or a "**", then it is matched only against the final component of the filename. (Remember that the algorithm is applied recursively so "full filename" can actually be any portion of a path from the starting directory on down

In my small test, this does work recursively down the directory tree and only selects the pdfs.