rsync list of files in sub-directories to remote server and delete extraneous

Question

I haven't been able to find the answer to this one. I have tried all kinds of combinations of --include, --exclude, -r, --delete-excluded, and source directory.

This is what I'm trying to accomplish:

• Source directory:

  Dir1/File1-1
  Dir1/File1-2
  Dir2/File2-1
  Dir2/File2-2
  Dir2/File2-3
  Dir3/File3-1
  Dir3/File3-2
  Dir3/File3-3
  ...
  

• Destination directory:
  (state unknown)

What I would like to do is to provide a pre-defined subset of files from the source directory with the relative path and rsync them to the remote destination, deleting anything on the destination (including directories) that wasn't in the subset of files.

For example, given the above Source Directory:

• My Pre-defined File List:

  Dir1/File1-2
  Dir3/File3-3
  

• Destination directory:

  Dir1/File1-1
  Dir1/File1-2
  Dir2/File2-1
  Dir3/File3-1
  

I would expect to have only Dir1/File1-2, Dir3/File3-3, and no Dir2 in the destination when I'm done. I will also NOT have expected Dir1/File1-2 to be copied over again.

Things to Note:

• These are very large files and I'm trying to prevent having to remotely copy them again if not needed.
• The contents of the source sub-directories changes everyday
• The pre-defined list changes everyday, but may/will include files in yesterday's list
• There are 0 files in the Source Directory. It only contains sub-directories which contain the files needed to sync
• The Sub-directories in the Source are pretty static. i.e. Dir1, Dir2 and Dir3 will always be there

I will keep plugging away to see if I can come up with the solution I'm looking for.

Update:

First of all, I want to thank jw013 for the formatting changes. I followed the help guideline, but didn't understand why is wasn't formatting correctly (first time poster).

Using the suggestions from Serge, I came up with the following command, but it still does not remove the extraneous files from the destination that are NOT in the File List.

• Command

  rsync -avRP --prune-empty-dirs --delete-delay --ignore-existing $FILE_LIST destination/
  

Answer 1

You can do this using

rsync -av --delete --update --include-from=LISTFILE   -f 'hide,! */' src/ dst

where LISTFILE is a file that contains all files that should be synchronized to the destination directory. If your files do not change, you can use --ignore-existing instead of --update. Read the rsync manual page for details.

Answer 2

There is a number of delete options you can use with rsync to delete files on the target if they are not exist in the source:

        --del                   an alias for --delete-during
        --delete                delete extraneous files from dest dirs
        --delete-before         receiver deletes before xfer, not during
        --delete-during         receiver deletes during the transfer
        --delete-delay          find deletions during, delete after
        --delete-after          receiver deletes after transfer, not during

Also, you may use --ignore-existing to skip copying the files that exist in the target:

        --ignore-existing       skip updating files that exist on receiver

And to delete the empty directories you should use:

    -m, --prune-empty-dirs      prune empty directory chains from file-list

So, the command

rsync -av --prune-empty-dirs --delete-delay --ignore-existing src/ dst

will do what you are looking for, I suppose

Answer 3

I think you need to focus on --delete-excluded. I've tried it locally and it seems to work as desired. I would start by trying

rsync -a --delete-excluded --include-from=LISTFILE --exclude="*" src/ dst

I'm not sure why certain other options are being suggested :P. The most "interesting" one is --ignore-existing. It may be worth making sure you understand the more friendly optimisations first - -a, or -u.

It looks like you're already using -a, which implies -t, which allows rsync to optimise by default. "Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time". If you're not using -a, it's often possible to use "-u or --update - skip files that are newer on the receiver".

If you want to benchmark the difference with the "quick check" optimisation disabled, you can use --checksum. (It's also useful if you want to forcibly resync all the existing files at some point in case of corruption. Enable -v and you'll be able to see which files had to be re-written).