rsync a list of directories with absolute path in text file

Question

I have a text file containing a list of directories with its absolute path

$ cat DirectoriesToCopy.txt

/data/Dir1
/data/Dir2

I want to use rsync to copy all these directories preserving its absolute path to another location.

I tried the following rsync command, but it doesn't work

rsync -avr --include-from=DirectoriesToCopy.txt --exclude='*/' --exclude='/*'  /  /media/MyDestination/

What is going wrong here?

Answer 1

I want to use rsync to copy all these directories [from a list] preserving its absolute path to another location

Don't use --include or --filter variations, as that will just confuse things. Instead, use --files-from=_filename_. (If you do, make sure you do lots of testing.)

1. Create the directory list in a file, one directory/file per line.

2. Use rsync's --files-from= with the above file.

3. Use --relative / -R option to make sure the source pathnames are copied at the end of the destination.

4. Even if you have the -a option, also include -r. From the man page:

   In both cases, if the -r option was enabled, that dir's entire hierarchy would also be transferred (keep in mind that -r needs to be specified explicitly with --files-from, since it is not implied by -a).

Complete command:

rsync ${DEBUG:+-nv} -arR --files-from=<list_of_files.txt> <top-level-dir>  <target-dir>

(the files in list_of_files.txt must be relative or found in top-level-dir)

(if DEBUG is set, rsync merely prints out what might have been copied.)

Answer 2

Use the following command:

rsync -av --include-from=DirectoriesToCopy.txt --include /data/ --exclude='/data/*' --exclude='/*/' / /media/MyDestination/

You need to include /data/ explicitly, you could also have added that to the list in the file. Then exclude all other directories (order is important with includes/excludes).

Note that your usage of -r was redundant as that's included in -a.

EDIT: You could also accomplish the same result with:

rsync -av --relative /data/Dir1 /data/Dir2 /media/MyDestination/

It's not rsync that's forcing you to do difficult things just to copy a couple of directories, it just gives you multiple ways of doing the same thing; in some cases going the include/exclude way may be more suited, here I'd do the --relative thing above (without --relative you'd end up with /media/MyDestination/Dir1 and /media/MyDestination/Dir2, with the --relative the whole source path is copied to the destination).

Answer 3

The rsync manual warns about this scenario (section “Include/exclude pattern rules”):

this won't work:

+ /some/path/this-file-will-not-be-found
+ /file-is-included
- *

This fails because the parent directory "some" is excluded by the '*' rule, so rsync never visits any of the files in the "some" or "some/path" directories. One solution is to ask for all directories in the hierarchy to be included by using a single rule: "+ */" (put it somewhere before the "- *" rule), and perhaps use the --prune-empty-dirs option. Another solution is to add specific include rules for all the parent dirs that need to be visited. For instance, this set of rules works fine:

+ /some/
+ /some/path/
+ /some/path/this-file-is-found
+ /file-also-included
- *

In your case, I think the simplest approach would be to preprocess the list of directories to include so that whenever you include /path/to/foo, you also include all the parent directories (/path/to, /path, /), and also include subdirectories of the original directories (/path/to/foo/***), and after all this have a rule that excludes everything not previously listed (*).

<DirectoriesToCopy.txt awk '
    {print "+ " $0 "/***"; while (sub(/\/+[^\/]+\/*$/, "/")) print "+ " $0}
    END {print "- *"}
' >rsync-rules.txt
rsync -avr --include-from=rsync-rules.txt  /  /media/MyDestination/