Recursively copying directories with a given name

Question

With a directory structure built using

#!/bin/bash
for name in A B
do
    mkdir -p /tmp/src/${name}/copyme
    echo "do not copy" > /tmp/src/${name}/no.txt
    echo "copy" > /tmp/src/${name}/copyme/yes.txt
done

I would like to mirror just the copyme directories, along with the files within them, to /tmp/tgt.

This should be simple. Relying on rsync distinguishing the order of the command-line options: exclude everything, then include the relevant pattern. Yet

rsync -av --exclude='*' --include='copyme' /tmp/src/ /tmp/tgt/

excludes everything (only the target directory is created). Why?

Answer 1

When rsync runs, it tests the found names at the source against the patterns, and the first matching pattern takes effect:

$ rsync -avv --exclude='*' --include='copyme' /tmp/src/ /tmp/tgt/
sending incremental file list
[sender] hiding directory A because of pattern *
[sender] hiding directory B because of pattern *
delta-transmission disabled for local transfer or --whole-file
total: matches=0  hash_hits=0  false_alarms=0 data=0

sent 51 bytes  received 86 bytes  274.00 bytes/sec
total size is 0  speedup is 0.00

When a directory has been excluded (see "hiding directory..." above), its content is not further considered. Reversing the exclusion and inclusion patterns will not help since it will never get to the copyme directories.

The rsync manual says:

For instance, to include /foo/bar/baz, the directories /foo and /foo/bar must not be excluded. Excluding one of those parent directories prevents the examination of its content, cutting off rsync's recursion into those paths and rendering the include for /foo/bar/baz ineffectual (since rsync can't match something it never sees in the cut-off section of the directory hierarchy).

So, instead:

$ rsync -avv --include='[AB]' --include='copyme/***' --exclude='*' /tmp/src/ /tmp/tgt/
sending incremental file list
[sender] showing directory A because of pattern [AB]
[sender] showing directory B because of pattern [AB]
[sender] showing directory A/copyme because of pattern copyme/***
[sender] hiding file A/no.txt because of pattern *
[sender] showing file A/copyme/yes.txt because of pattern copyme/***
[sender] showing directory B/copyme because of pattern copyme/***
[sender] hiding file B/no.txt because of pattern *
[sender] showing file B/copyme/yes.txt because of pattern copyme/***
created directory /tmp/tgt
delta-transmission disabled for local transfer or --whole-file
./
A/
A/copyme/
A/copyme/yes.txt
B/
B/copyme/
B/copyme/yes.txt
total: matches=0  hash_hits=0  false_alarms=0 data=10

sent 305 bytes  received 175 bytes  960.00 bytes/sec
total size is 10  speedup is 0.02

Note that the exclusion has to come last, after the inclusions. The copyme/*** pattern matches both the copyme directory name itself and any pathname underneath it.

If you don't want to hardcode the A and B directory names:

for dir in /tmp/src/*; do
    [ ! -d "$dir" ] && continue
    rsync -avv --include="${dir##*/}" --include='copyme/***' --exclude='*' /tmp/src/ /tmp/tgt/
done

This would output

sending incremental file list
[sender] showing directory A because of pattern A
[sender] hiding directory B because of pattern *
[sender] showing directory A/copyme because of pattern copyme/***
[sender] hiding file A/no.txt because of pattern *
[sender] showing file A/copyme/yes.txt because of pattern copyme/***
created directory /tmp/tgt
delta-transmission disabled for local transfer or --whole-file
./
A/
A/copyme/
A/copyme/yes.txt
total: matches=0  hash_hits=0  false_alarms=0 data=5

sent 180 bytes  received 148 bytes  656.00 bytes/sec
total size is 5  speedup is 0.02
sending incremental file list
[sender] hiding directory A because of pattern *
[sender] showing directory B because of pattern B
[sender] showing directory B/copyme because of pattern copyme/***
[sender] hiding file B/no.txt because of pattern *
[sender] showing file B/copyme/yes.txt because of pattern copyme/***
delta-transmission disabled for local transfer or --whole-file
B/
B/copyme/
B/copyme/yes.txt
total: matches=0  hash_hits=0  false_alarms=0 data=5

sent 180 bytes  received 117 bytes  594.00 bytes/sec
total size is 5  speedup is 0.02

The result would be

$ tree src tgt
src
|-- A
|   |-- copyme
|   |   `-- yes.txt
|   `-- no.txt
`-- B
    |-- copyme
    |   `-- yes.txt
    `-- no.txt

4 directories, 4 files
tgt
|-- A
|   `-- copyme
|       `-- yes.txt
`-- B
    `-- copyme
        `-- yes.txt

4 directories, 2 files

---

Another approach which does not use exclusion or inclusion patterns with rsync but that uses find to locate the copyme directories and then rsync to copy them:

find /tmp/src -type d -name 'copyme' -prune -exec sh -c '
    cd /tmp/src && rsync -aRvv "${1#/tmp/src/}/" /tmp/tgt/' sh {} ';'

Notice the -R (--relative) flag used with rsync here.

The sh -c script executed for each found copyme directory does a cd to /tmp/src and then copies the pathname with the initial /tmp/src bit of its path removed.

The -prune in the find command stops find from looking for further copyme directories inside the found directory.