8

I have a text file containing a list of directories with its absolute path

$ cat DirectoriesToCopy.txt

/data/Dir1
/data/Dir2

I want to use rsync to copy all these directories preserving its absolute path to another location.

I tried the following rsync command, but it doesn't work

rsync -avr --include-from=DirectoriesToCopy.txt --exclude='*/' --exclude='/*'  /  /media/MyDestination/

What is going wrong here?

  • Do you want to transfer /data/Dir1 and /data/Dir2 including their contents to /media/MyDestination/Dir1 and /media/MyDestination/Dir2 ? Or do you want to include the /data part? How large is this list, is it too big to simply list those entries on the command line? Anyway, your --exclude statements prevent the recursion. – wurtel Dec 17 '14 at 14:40
  • I want to transfer all the files inside those directories as well. There are many files in that directory, so I would like to avoid creating another text file with all the filenames. – indiajoe Dec 17 '14 at 14:56
  • I would like to keep my full path like /media/MyDestination/data/Dir2/ . I added the --exclude to prevent other directories and files in / from getting copied. – indiajoe Dec 17 '14 at 15:00
  • In cases where the list of directories are not very large, i guess the following option is more easier rsync -av $(cat DirectotiesToCopy.txt) /media/MyDestination/ – indiajoe Dec 18 '14 at 4:40
6

Use the following command:

rsync -av --include-from=DirectoriesToCopy.txt --include /data/ --exclude='/data/*' --exclude='/*/' / /media/MyDestination/

You need to include /data/ explicitly, you could also have added that to the list in the file. Then exclude all other directories (order is important with includes/excludes).

Note that your usage of -r was redundant as that's included in -a.

EDIT: You could also accomplish the same result with:

rsync -av --relative /data/Dir1 /data/Dir2 /media/MyDestination/

It's not rsync that's forcing you to do difficult things just to copy a couple of directories, it just gives you multiple ways of doing the same thing; in some cases going the include/exclude way may be more suited, here I'd do the --relative thing above (without --relative you'd end up with /media/MyDestination/Dir1 and /media/MyDestination/Dir2, with the --relative the whole source path is copied to the destination).

  • Thank you that worked. I am a little surprised that one has to define such careful sequence of filters for simply copying a list of directories using rsync. For example, if my list of directories were not under one common directory /data/ but was instead spread around many locations, the filters to define becomes quite complicated. – indiajoe Dec 17 '14 at 16:33
  • See my edit to my answer. – wurtel Dec 18 '14 at 7:53
  • Thanks. So if my list of directories is not very huge, I could use the --relative flag as shown below. rsync -av --relative $(cat DirectotiesToCopy.txt) /media/MyDestination/ – indiajoe Dec 19 '14 at 2:50
  • Yes; keep in mind --relative is just another option. – wurtel Dec 22 '14 at 10:36
7

I want to use rsync to copy all these directories [from a list] preserving its absolute path to another location

Don't use --include or --filter variations, as that will just confuse things. Instead, use --files-from=_filename_. (If you do, make sure you do lots of testing.)

  1. Create the directory list in a file, one directory/file per line.

  2. Use rsync's --files-from= with the above file.

  3. Use --relative / -R option to make sure the source pathnames are copied at the end of the destination.

  4. Even if you have the -a option, also include -r. From the man page:

    In both cases, if the -r option was enabled, that dir's entire hierarchy would also be transferred (keep in mind that -r needs to be specified explicitly with --files-from, since it is not implied by -a).

Complete command:

rsync ${DEBUG:+-nv} -arR --files-from=<list_of_files.txt> <top-level-dir>  <target-dir>

(the files in list_of_files.txt must be relative or found in top-level-dir)

(if DEBUG is set, rsync merely prints out what might have been copied.)

  • 1
    This is what I needed! However "--include-from" should be "--files-from". Also I used "--list-only" to see a preview of what RSYNC will do before actually letting it work. – Adambean Feb 3 '17 at 11:52
  • 1
    Thanks, I've updated the answer. --include-from will also work, but then what I said about -r is largely irrelevant. – Otheus Feb 7 '17 at 16:17
2

The rsync manual warns about this scenario (section “Include/exclude pattern rules”):

this won't work:

+ /some/path/this-file-will-not-be-found
+ /file-is-included
- *

This fails because the parent directory "some" is excluded by the '*' rule, so rsync never visits any of the files in the "some" or "some/path" directories. One solution is to ask for all directories in the hierarchy to be included by using a single rule: "+ */" (put it somewhere before the "- *" rule), and perhaps use the --prune-empty-dirs option. Another solution is to add specific include rules for all the parent dirs that need to be visited. For instance, this set of rules works fine:

+ /some/
+ /some/path/
+ /some/path/this-file-is-found
+ /file-also-included
- *

In your case, I think the simplest approach would be to preprocess the list of directories to include so that whenever you include /path/to/foo, you also include all the parent directories (/path/to, /path, /), and also include subdirectories of the original directories (/path/to/foo/***), and after all this have a rule that excludes everything not previously listed (*).

<DirectoriesToCopy.txt awk '
    {print "+ " $0 "/***"; while (sub(/\/+[^\/]+\/*$/, "/")) print "+ " $0}
    END {print "- *"}
' >rsync-rules.txt
rsync -avr --include-from=rsync-rules.txt  /  /media/MyDestination/
  • Thanks for the detailed explanation and script to create the rules file. I understand the issue now. I still feel it would have been nice for rsync to have an option like --dirs-from=DirectoriesToCopy.txt as a short cut to all the complicated filters. – indiajoe Dec 18 '14 at 4:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.