23

I'm using rsync to copy some files from a share to another.

Recursively, I need to:

  • Delete files at destination that were removed from the origin
  • Only sync .php and .js files
  • Exclude every other file type
  • Do not delete .svn/ directories at destination

If I use this:

rsync -zavC --delete --include='*.php' --include='*.js' --exclude="*" \
    /origin /destination

Then rsync is not recursive because exclude="*" excludes all files but also folders.

If I add --include="*/" then the .svn/ directory gets deleted (it also gets included).

How can I solve this mind blasting dilemma?

uname -a:

Linux tux 3.9.2-1-ARCH #1 SMP PREEMPT Sat May 11 20:31:08 CEST 2013 x86_64 GNU/Linux

rsync version:

rsync 3.0.9-6

3 Answers 3

14

1st attempt (didn't work)

You need to include the directories in addition to the files:

rsync -zavC --delete --include '*/' --include='*.php' --include='*.js' \
     --exclude="*" /media/datacod/Test/ /home/lucas/Desktop/rsync/

2nd attempt

rsync -avzC --filter='-rs_*/.svn*' --include="*/" --include='*.js' \
     --include='*.php' --exclude="*" --delete dir1/ dir2/

test data

I wrote this script to create some sample data to test this out on. Here's that script, setup_svn_sample.bash:

#!/bin/bash

# setup .svn dirs
mkdir -p dir{1,2}/dir{1,2,3,4}/.svn

# fake data under .svn
mkdir -p dir1/dir{1,2,3,4}/.svn/origdir
mkdir -p dir2/dir{1,2,3,4}/.svn/keepdir

# files to not sync
touch dir1/dir{1,2,3,4}/file{1,2}

# files to sync
touch dir1/dir{1,2,3,4}/file1.js
touch dir1/dir{1,2,3,4}/file1.php

Running it produces the following directories:

source dir

$ tree -a dir1
dir1
|-- dir1
|   |-- file1
|   |-- file1.js
|   |-- file1.php
|   |-- file2
|   `-- .svn
|       `-- origdir
|-- dir2
|   |-- file1
|   |-- file1.js
|   |-- file1.php
|   |-- file2
|   `-- .svn
|       `-- origdir
|-- dir3
|   |-- file1
|   |-- file1.js
|   |-- file1.php
|   |-- file2
|   `-- .svn
|       `-- origdir
`-- dir4
    |-- file1
    |-- file1.js
    |-- file1.php
    |-- file2
    `-- .svn
        `-- origdir

destination dir

$ tree -a dir2
dir2
|-- dir1
|   `-- .svn
|       `-- keepdir
|-- dir2
|   `-- .svn
|       `-- keepdir
|-- dir3
|   `-- .svn
|       `-- keepdir
`-- dir4
    `-- .svn
        `-- keepdir

Running the above rsync command:

rsync -avzC --filter='-rs_*/.svn*' --include="*/" --include='*.js' \
     --include='*.php' --exclude="*" --delete dir1/ dir2/
sending incremental file list
dir1/file1.js
dir1/file1.php
dir2/file1.js
dir2/file1.php
dir3/file1.js
dir3/file1.php
dir4/file1.js
dir4/file1.php

sent 480 bytes  received 168 bytes  1296.00 bytes/sec
total size is 0  speedup is 0.00

Resulting dir2 afterards:

$ tree -a dir2
dir2
|-- dir1
|   |-- file1.js
|   |-- file1.php
|   `-- .svn
|       `-- keepdir
|-- dir2
|   |-- file1.js
|   |-- file1.php
|   `-- .svn
|       `-- keepdir
|-- dir3
|   |-- file1.js
|   |-- file1.php
|   `-- .svn
|       `-- keepdir
`-- dir4
    |-- file1.js
    |-- file1.php
    `-- .svn
        `-- keepdir

Why does it work?

The key piece to this script is to make use of the filters capability of rsync. Filters allow you to remove files from the matched set at various points in the command. So in our case we're filtering any files that match the pattern */.svn*. The modifiers -rs_ tell the filter that we want to filter on both the source side as well as the target side.

excerpt from the FILTER NOTES section of rsync's man page

  • An s is used to indicate that the rule applies to the sending side. When a rule affects the sending side, it prevents files from being transferred. The default is for a rule to affect both sides unless --delete-excluded was specified, in which case default rules become sender-side only. See also the hide (H) and show (S) rules, which are an alternate way to specify sending-side includes/excludes.

  • An r is used to indicate that the rule applies to the receiving side. When a rule affects the receiving side, it prevents files from being deleted. See the s modifier for more info. See also the protect (P) and risk (R) rules, which are an alternate way to specify receiver-side includes/excludes.

See man rsync for more details.

Tips for figuring this out (hint using --dry-run)

While describing how to do this I thought I'd mention the --dry-run switch to rsync. It' extremely useful in seeing what will happen without having the rsync actually take place.

For Example

Using the following command will do a test run and show us the decision logic behind rsync:

rsync --dry-run -avvzC --filter='-rs_*/.svn*' --include="*/" \
     --include='*.js' --include='*.php' --exclude="*" --delete dir1/ dir2/
sending incremental file list
[sender] showing directory dir3 because of pattern */
[sender] showing directory dir2 because of pattern */
[sender] showing directory dir4 because of pattern */
[sender] showing directory dir1 because of pattern */
[sender] hiding file dir1/file1 because of pattern *
[sender] showing file dir1/file1.js because of pattern *.js
[sender] hiding file dir1/file2 because of pattern *
[sender] showing file dir1/file1.php because of pattern *.php
[sender] hiding directory dir1/.svn because of pattern */.svn*
[sender] hiding file dir2/file1 because of pattern *
[sender] showing file dir2/file1.js because of pattern *.js
[sender] hiding file dir2/file2 because of pattern *
[sender] showing file dir2/file1.php because of pattern *.php
[sender] hiding directory dir2/.svn because of pattern */.svn*
[sender] hiding file dir3/file1 because of pattern *
[sender] showing file dir3/file1.js because of pattern *.js
[sender] hiding file dir3/file2 because of pattern *
[sender] showing file dir3/file1.php because of pattern *.php
[sender] hiding directory dir3/.svn because of pattern */.svn*
[sender] hiding file dir4/file1 because of pattern *
[sender] showing file dir4/file1.js because of pattern *.js
[sender] hiding file dir4/file2 because of pattern *
[sender] showing file dir4/file1.php because of pattern *.php
[sender] hiding directory dir4/.svn because of pattern */.svn*
delta-transmission disabled for local transfer or --whole-file
[generator] risking directory dir3 because of pattern */
[generator] risking directory dir2 because of pattern */
[generator] risking directory dir4 because of pattern */
[generator] risking directory dir1 because of pattern */
[generator] protecting directory dir1/.svn because of pattern */.svn*
dir1/file1.js
dir1/file1.php
[generator] protecting directory dir2/.svn because of pattern */.svn*
dir2/file1.js
dir2/file1.php
[generator] protecting directory dir3/.svn because of pattern */.svn*
dir3/file1.js
dir3/file1.php
[generator] protecting directory dir4/.svn because of pattern */.svn*
dir4/file1.js
dir4/file1.php
total: matches=0  hash_hits=0  false_alarms=0 data=0

sent 231 bytes  received 55 bytes  572.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)

In the above output you can see that the ./svn directories are being protected by our filter rule. Valuable insight for debugging the rsync.

References

3
  • Already tried this, as I said above. If I do this, then the .SVN directory gets also included (most likely deleted, because It doesnt exist in the origin, or modified). It is critical to leave the .SVN directory untouched. Thanks for the try anyway! :)
    – canolucas
    May 17, 2013 at 20:04
  • -C is supposed to ignore ".SVN/", but include="*/" includes it
    – canolucas
    May 17, 2013 at 20:07
  • You are right. It should be .svn, editing the answer. Anyway, after renaming the issue remains. -C and include="*/" don't seem to be very close friends :(
    – canolucas
    May 17, 2013 at 20:27
10

Ok, after several tries I sorted this up:

rsync -vaiz --delete --exclude=.svn/ --include='*.php' --include='*.js' \
    --include='*/' --exclude='*' --prune-empty-dirs \
    --filter "protect .svn/" /origin /destination

Thank you

3
  • Great answer. My filter protects .svn/ dirs from --prune-empty-dirs in the case they are empty. Another approach, this one duplicating empty dirs: rsync -vaiz --delete --exclude=.svn/ --include='*.php' --include='*.js' --include='*/' --exclude='*' /origin /destination
    – canolucas
    May 18, 2013 at 0:22
  • The key of my approach is to --exclude=.svn/ before the --include's
    – canolucas
    May 18, 2013 at 0:23
  • Good answer as well. I would've suggested a exclude='.svn/' but I thought you wanted to use the -C. Glad you figured it out!
    – slm
    May 18, 2013 at 1:32
0

Although this is not using rsync another option is to use find and cpio. For example I have a directory called Photos and that the year and month.

Photos
├── 2002
│   ├── 2002-03
│   │   ├── 2002-03-30
│   │   ├── 2002-03-31
│   │   └── 2002-03-31-02
│   ├── 2002-04
│   │   ├── 2002-04-01
│   │   ├── 2002-04-01-03
│   │   ├── 2002-04-07
│   │   ├── 2002-04-21
│   │   ├── 2002-04-22
│   │   ├── 2002-04-22-02
│   │   └── 2002-04-27
│   ├── 2002-05
│   │   ├── 2002-05-02
│   │   ├── 2002-05-03

But in here I have jpg, dng, xml and other things and I want just the jpg's so I would do this

"cd" to the directory above the Photos directory then:

find Photos -type f -name "*jpg" -print | cpio -pdmvu /fast

And off it goes

/fast/Photos/2002/2002-04/2002-04-22/bath problem.jpg
/fast/Photos/2002/2002-04/2002-04-22-02/full bath.jpg
/fast/Photos/2002/2002-07/2002-07-10/Broken Top.jpg
/fast/Photos/2002/2002-12/2002-12-28/101-0125_IM~~G.jpg
/fast/Photos/2002/2002-12/2002-12-28/small-101-0125_IM~~G.jpg
/fast/Photos/2003/2003-01/2003-01-19/k1.jpg
/fast/Photos/2003/2003-01/2003-01-19/k2.jpg
/fast/Photos/2003/2003-02/2003-02-23/quinn.jpg
/fast/Photos/2003/2003-05/2003-05-04/all.jpg
/fast/Photos/2003/2003-05/2003-05-09/100_0015_r1.jpg
/fast/Photos/2003/2003-05/2003-05-09/100_0006_2_r1.jpg
/fast/Photos/2003/2003-05/2003-05-09/100_0006_r1.jpg
/fast/Photos/2003/2003-05/2003-05-09/100_0007_2_r1.jpg

You can use this method for lots of things, years ago this is how we would copy the / filesystem when moving to larger disks as it would copy device files.

2
  • That doesn't answer the question.
    – RalfFriedl
    Sep 22, 2019 at 14:10
  • It does if he uses: find /origin -type f ( -name ".php" -o -name ".js" ) -print | cpio -pdmvu /destination Only the *.php and *.js files will be in the target directory.
    – lxtwin
    Sep 23, 2019 at 9:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .