2

I have a directory with the following structure:

-- 201893208
   └── 8Z12
          └── ko_8Z12_Full
          └── wp_we_8Z12_FullDAT
          └── 8Z12_DATFull
   └── P011
          └── P011_Full
          └── 8Z12_FullDAT
          └── P011_DATFull
   └── 9FZA
          └── kl_wt-we-w_kl9-9FZA_Full
          └── ffd-9FZA_FullDAT
          └── 8fdZ12232_9FZA_DATFull
-- 903240920
   └── P0fsa
          └── P0fsa_Full
          └── P0fsa_FullDAT
          └── P0fsa_DATFull
   └── Paaaf
          └── we-Paaaf_ww_fl_Full
          └── Paaaf_FullDAT
          └── Paaaf_DATFull
   └── 9FZATYYY
          └── 9FZATYYY_Full
          └── 9FZATYYY_FullDAT
-- wt0340291
   └── OPF1121
          └── OPF1121_Full
          └── 8Z12_DATFull
   └── KLOFJ9
          └── lop_KLOFJ9_ffj_Full
          └── powt_KLOFJ9_DATFull
   └── LP02323
          └── wr_we_LP02323_Full
          └── wr_we_LP02323_FullDAT

There are thousands of files in each and every folder listed above. And then there are also many subdirectories inside each. For example 8Z12 does not contain only the three folders listed above, but it also contains thousands of files.

I ONLY want to copy directories which have _Full at the end of their name (DATFull directories should NOT be copied) AND also contain the patterns in the list below

LP02323
KLOFJ9
Paaaf
9FZA

In other words, the directories that contain a string from the list above AND Full in their name (but NOT DAT) should be copied.

Therefore in the example above, only the following directories (and all of their contents and sub-directories) should be copied:

wr_we_LP02323_Full
lop_KLOFJ9_ffj_Full
we-Paaaf_ww_fl_Full
kl_wt-we-w_kl9-9FZA_Full

To my understanding, rsync does not support regex, therefore this has to be done with find first of all (correct me if I am wrong). But how can I do this in a way to ensure all of the directories and sub-directories are checked and all of the relevant folders are copied even if they are buried very deep in several sub-directories (note that the above example was a simplified structure of my original folder).

So there are two questions:

  • How can I provide the list of patterns to find?
  • How can I pipe the result of find to rsync?

So far this only comes to my mind to match Full:

find . -regextype sed -regex ".*/.*[^DAT]Full$"

But how can I add the list of IDs to this find command?

0

Using a file of strings that need to be present in the directory names, a shell loop over these, and rsync (assuming we want to copy from the directory in the variable $source and to the directory in the variable $target):

while IFS= read -r string; do
    rsync --archive --exclude='*DAT*/' --include='*/' --include="*$string*_Full/***" --exclude='*' \
        --prune-empty-dirs "$source"/ "$target"
done <strings.txt

What the options to rsync does (the first hit on any of the exclusion/inclusion patters matters):

  • --archive: Copies ownership, permissions, timestamps etc.
  • --exclude='*DAT*/': Exclude any directory with DAT in its name.
  • --include='*/': Consider all directories (except for the ones excluded by the previous pattern). This is needed for rsync to reach the actual directories that you are interested in.
  • --include="*$string*_Full/***": Consider all directories matching the given pattern and everything below that directory. If $string is parrot, this would be --include="*parrot*_Full/***".
  • --exclude='*': Don't consider anything not already explicitly included.
  • --prune-empty-dirs: Don't transfer directories that doesn't have anything explicitly included in them.

If you want to see how rsync evaluates the patterns when you run it, add -vv to the rsync command line.

Testing:

$ tree
.
|-- from
|   `-- a
|       `-- b
|           |-- c_A_DATFull
|           |   `-- file
|           |-- c_A_DAT_Full
|           |   `-- file
|           |-- c_A_Full
|           |   `-- file
|           |-- c_B_DATFull
|           |   `-- file
|           |-- c_B_DAT_Full
|           |   `-- file
|           |-- c_B_Full
|           |   `-- file
|           |-- c_C_DATFull
|           |   `-- file
|           |-- c_C_DAT_Full
|           |   `-- file
|           `-- c_C_Full
|               `-- file
`-- strings.txt

12 directories, 10 files

$ cat strings.txt
A
B

$ source=from
$ target=to

(running the loop here)

$ tree
.
|-- from
[...]
`-- to
    `-- a
        `-- b
            |-- c_A_Full
            |   `-- file
            `-- c_B_Full
                `-- file

17 directories, 12 files

With a single invocation of rsync:

set -- --exclude='*DAT*/' --include='*/'
while IFS= read -r string; do
    set -- "$@" --include="*$string*_Full/***"
done <strings.txt
set -- "$@" --exclude='*'

rsync --archive "$@" --prune-empty-dirs "$source"/ "$target"

A find solution:

set --
while IFS= read -r string; do
    set -- "$@" -o -name "*$string*_Full"
done <strings.txt
shift

# "$@" would now be something like
#    -name *LP02323*_Full -o -name *Paaaf*_Full -o -name ...etc

find "$source" -type d '(' "$@" ')' ! -name '*DAT*' -exec sh -c '
    source=$1; target=$2; shift 2
    for pathname do
        mkdir -p "$target/${pathname#$source}"
        rsync --archive "$pathname"/ "$target/${pathname#$source}"
    done' sh "$source" "$target" {} +

This would use find to generate a list of the subdirectories that you would like to copy. These are given to a small inline script that loops over them.

In each iteration of the loop, the corresponding directory on the target is created (this assumes local copy), and the directory is copied using rsync.

Never ever use find to pipe pathnames to some other command unless you can arrange for the pathnames to be safely delimited.

Related:

  • 1
    @benett rsync does not do regular expressions, but filename globbing patterns (with some extentions, such as ***). You are rigt that that directory would be copied. I thought DAT always occurred as DATFull at the end. This is easily fixed. I will update very soon. – Kusalananda Jul 11 '18 at 9:57
  • 1
    @benett Fixed now. – Kusalananda Jul 11 '18 at 10:00
  • 1
    @benett Sorry, I got the ordering of the inclusion/exclusion patterns wrong. Sorted now. – Kusalananda Jul 11 '18 at 10:04
  • 1
    @benett One string per line, yes. I'll make a small adjustment to the read calls to be able to read the newline delimited strings safely. – Kusalananda Jul 11 '18 at 11:38
  • 1
    @benett The approach with find is only able to cope with local copying unfortunately. You can not run find on a remote host. If your source or target accesses a remote host, then find can not easily be used. – Kusalananda Jul 11 '18 at 11:57

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.