Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I haven't been able to find the answer to this one. I have tried all kinds of combinations of --include, --exclude, -r, --delete-excluded, and source directory.

This is what I'm trying to accomplish:

  • Source directory:

    Dir1/File1-1
    Dir1/File1-2
    Dir2/File2-1
    Dir2/File2-2
    Dir2/File2-3
    Dir3/File3-1
    Dir3/File3-2
    Dir3/File3-3
    ...
    
  • Destination directory:
    (state unknown)

What I would like to do is to provide a pre-defined subset of files from the source directory with the relative path and rsync them to the remote destination, deleting anything on the destination (including directories) that wasn't in the subset of files.

For example, given the above Source Directory:

  • My Pre-defined File List:

    Dir1/File1-2
    Dir3/File3-3
    
  • Destination directory:

    Dir1/File1-1
    Dir1/File1-2
    Dir2/File2-1
    Dir3/File3-1
    

I would expect to have only Dir1/File1-2, Dir3/File3-3, and no Dir2 in the destination when I'm done. I will also NOT have expected Dir1/File1-2 to be copied over again.

Things to Note:

  • These are very large files and I'm trying to prevent having to remotely copy them again if not needed.
  • The contents of the source sub-directories changes everyday
  • The pre-defined list changes everyday, but may/will include files in yesterday's list
  • There are 0 files in the Source Directory. It only contains sub-directories which contain the files needed to sync
  • The Sub-directories in the Source are pretty static. i.e. Dir1, Dir2 and Dir3 will always be there

I will keep plugging away to see if I can come up with the solution I'm looking for.

Update:

First of all, I want to thank jw013 for the formatting changes. I followed the help guideline, but didn't understand why is wasn't formatting correctly (first time poster).

Using the suggestions from Serge, I came up with the following command, but it still does not remove the extraneous files from the destination that are NOT in the File List.

  • Command

    rsync -avRP --prune-empty-dirs --delete-delay --ignore-existing $FILE_LIST destination/
    
share|improve this question

3 Answers 3

There is a number of delete options you can use with rsync to delete files on the target if they are not exist in the source:

        --del                   an alias for --delete-during
        --delete                delete extraneous files from dest dirs
        --delete-before         receiver deletes before xfer, not during
        --delete-during         receiver deletes during the transfer
        --delete-delay          find deletions during, delete after
        --delete-after          receiver deletes after transfer, not during

Also, you may use --ignore-existing to skip copying the files that exist in the target:

        --ignore-existing       skip updating files that exist on receiver

And to delete the empty directories you should use:

    -m, --prune-empty-dirs      prune empty directory chains from file-list

So, the command

rsync -av --prune-empty-dirs --delete-delay --ignore-existing src/ dst

will do what you are looking for, I suppose

share|improve this answer

You can do this using

rsync -av --delete --update --include-from=LISTFILE   -f 'hide,! */' src/ dst

where LISTFILE is a file that contains all files that should be synchronized to the destination directory. If your files do not change, you can use --ignore-existing instead of --update. Read the rsync manual page for details.

share|improve this answer

I think you need to focus on --delete-excluded. I've tried it locally and it seems to work as desired. I would start by trying

rsync -a --delete-excluded --include-from=LISTFILE --exclude="*" src/ dst

I'm not sure why certain other options are being suggested :P. The most "interesting" one is --ignore-existing. It may be worth making sure you understand the more friendly optimisations first - -a, or -u.

It looks like you're already using -a, which implies -t, which allows rsync to optimise by default. "Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time". If you're not using -a, it's often possible to use "-u or --update - skip files that are newer on the receiver".

If you want to benchmark the difference with the "quick check" optimisation disabled, you can use --checksum. (It's also useful if you want to forcibly resync all the existing files at some point in case of corruption. Enable -v and you'll be able to see which files had to be re-written).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.