12

I have a text file containing a list of directories with its absolute path

$ cat DirectoriesToCopy.txt

/data/Dir1
/data/Dir2

I want to use rsync to copy all these directories preserving its absolute path to another location.

I tried the following rsync command, but it doesn't work

rsync -avr --include-from=DirectoriesToCopy.txt --exclude='*/' --exclude='/*'  /  /media/MyDestination/

What is going wrong here?

4
  • Do you want to transfer /data/Dir1 and /data/Dir2 including their contents to /media/MyDestination/Dir1 and /media/MyDestination/Dir2 ? Or do you want to include the /data part? How large is this list, is it too big to simply list those entries on the command line? Anyway, your --exclude statements prevent the recursion.
    – wurtel
    Commented Dec 17, 2014 at 14:40
  • I want to transfer all the files inside those directories as well. There are many files in that directory, so I would like to avoid creating another text file with all the filenames.
    – indiajoe
    Commented Dec 17, 2014 at 14:56
  • I would like to keep my full path like /media/MyDestination/data/Dir2/ . I added the --exclude to prevent other directories and files in / from getting copied.
    – indiajoe
    Commented Dec 17, 2014 at 15:00
  • In cases where the list of directories are not very large, i guess the following option is more easier rsync -av $(cat DirectotiesToCopy.txt) /media/MyDestination/
    – indiajoe
    Commented Dec 18, 2014 at 4:40

3 Answers 3

10

I want to use rsync to copy all these directories [from a list] preserving its absolute path to another location

Don't use --include or --filter variations, as that will just confuse things. Instead, use --files-from=_filename_. (If you do, make sure you do lots of testing.)

  1. Create the directory list in a file, one directory/file per line.

  2. Use rsync's --files-from= with the above file.

  3. Use --relative / -R option to make sure the source pathnames are copied at the end of the destination.

  4. Even if you have the -a option, also include -r. From the man page:

    In both cases, if the -r option was enabled, that dir's entire hierarchy would also be transferred (keep in mind that -r needs to be specified explicitly with --files-from, since it is not implied by -a).

Complete command:

rsync ${DEBUG:+-nv} -arR --files-from=<list_of_files.txt> <top-level-dir>  <target-dir>

(the files in list_of_files.txt must be relative or found in top-level-dir)

(if DEBUG is set, rsync merely prints out what might have been copied.)

2
  • 1
    This is what I needed! However "--include-from" should be "--files-from". Also I used "--list-only" to see a preview of what RSYNC will do before actually letting it work.
    – Adambean
    Commented Feb 3, 2017 at 11:52
  • 1
    Thanks, I've updated the answer. --include-from will also work, but then what I said about -r is largely irrelevant.
    – Otheus
    Commented Feb 7, 2017 at 16:17
7

Use the following command:

rsync -av --include-from=DirectoriesToCopy.txt --include /data/ --exclude='/data/*' --exclude='/*/' / /media/MyDestination/

You need to include /data/ explicitly, you could also have added that to the list in the file. Then exclude all other directories (order is important with includes/excludes).

Note that your usage of -r was redundant as that's included in -a.

EDIT: You could also accomplish the same result with:

rsync -av --relative /data/Dir1 /data/Dir2 /media/MyDestination/

It's not rsync that's forcing you to do difficult things just to copy a couple of directories, it just gives you multiple ways of doing the same thing; in some cases going the include/exclude way may be more suited, here I'd do the --relative thing above (without --relative you'd end up with /media/MyDestination/Dir1 and /media/MyDestination/Dir2, with the --relative the whole source path is copied to the destination).

6
  • Thank you that worked. I am a little surprised that one has to define such careful sequence of filters for simply copying a list of directories using rsync. For example, if my list of directories were not under one common directory /data/ but was instead spread around many locations, the filters to define becomes quite complicated.
    – indiajoe
    Commented Dec 17, 2014 at 16:33
  • See my edit to my answer.
    – wurtel
    Commented Dec 18, 2014 at 7:53
  • Thanks. So if my list of directories is not very huge, I could use the --relative flag as shown below. rsync -av --relative $(cat DirectotiesToCopy.txt) /media/MyDestination/
    – indiajoe
    Commented Dec 19, 2014 at 2:50
  • Yes; keep in mind --relative is just another option.
    – wurtel
    Commented Dec 22, 2014 at 10:36
  • I would use $(cat DirectoriesToCopy.txt | xargs) instead to make sure the arguments that are one on each line are actually passed as arguments to rsync
    – leroyse
    Commented Jun 25, 2021 at 15:24
4

The rsync manual warns about this scenario (section “Include/exclude pattern rules”):

this won't work:

+ /some/path/this-file-will-not-be-found
+ /file-is-included
- *

This fails because the parent directory "some" is excluded by the '*' rule, so rsync never visits any of the files in the "some" or "some/path" directories. One solution is to ask for all directories in the hierarchy to be included by using a single rule: "+ */" (put it somewhere before the "- *" rule), and perhaps use the --prune-empty-dirs option. Another solution is to add specific include rules for all the parent dirs that need to be visited. For instance, this set of rules works fine:

+ /some/
+ /some/path/
+ /some/path/this-file-is-found
+ /file-also-included
- *

In your case, I think the simplest approach would be to preprocess the list of directories to include so that whenever you include /path/to/foo, you also include all the parent directories (/path/to, /path, /), and also include subdirectories of the original directories (/path/to/foo/***), and after all this have a rule that excludes everything not previously listed (*).

<DirectoriesToCopy.txt awk '
    {print "+ " $0 "/***"; while (sub(/\/+[^\/]+\/*$/, "/")) print "+ " $0}
    END {print "- *"}
' >rsync-rules.txt
rsync -avr --include-from=rsync-rules.txt  /  /media/MyDestination/
1
  • Thanks for the detailed explanation and script to create the rules file. I understand the issue now. I still feel it would have been nice for rsync to have an option like --dirs-from=DirectoriesToCopy.txt as a short cut to all the complicated filters.
    – indiajoe
    Commented Dec 18, 2014 at 4:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .