Use variable to exclude multiple directories with rsync

Question

The following snippet of code is part of a larger Bash script I'm working on. My intention is to pass input variables as arguments to Rsync.

I managed to make most of it work, except for the "--exclude" parameter's arguments:

INCLUDE="/.vim*,/.git*,/.ssh,.bash_aliases,.bashrc"  

rsync ${DRYRUNARG} -avzhP --delete ${SOURCE_DIRS} \                                                 
    --include=${INCLUDE} \                                                                          
    --exclude={/VirtualBox_VMs/*/Snapshots,/snap,/Downloads,/.*} \                                  
    ${MNTPNT}${DEST_DIR}

This code works perfectly for what I need, but I'd still like to see those directories that have to be excluded defined by a variable.

The resulting code that I need should be something like this:

INCLUDE="/.vim*,/.git*,/.ssh,.bash_aliases,.bashrc"
EXCLUDE="/VirtualBox_VMs/*/Snapshots,/snap,/Downloads,/.*"       

rsync ${DRYRUNARG} -avzhP --delete ${SOURCE_DIRS} \                                                 
    --include=${INCLUDE} \                                                                          
    --exclude=${EXCLUDE} \                                  
    ${MNTPNT}${DEST_DIR}

The code above doesn't work. The command executes without error but fails to exclude anything. The funny part is that it does work if I define the EXCLUDE variable with just one directory, even though it follows the same structure used for the INCLUDE variable.

I'm sure it's some silly mistake I'm making. Where could it be?

Answer 1

Do not demote a list of separate entries to a single string. This makes it difficult to parse out the different entries and you have to deal with the issues of entries potentially containing the character used as the delimiter, etc. Also, rsync does not know how to parse comma-delimited lists of include/exclude patterns.

Instead, use arrays:

#!/bin/bash

INCLUDE=( '/.vim*' '/.git*' /.ssh .bash_aliases .bashrc )
EXCLUDE=( '/VirtualBox_VMs/*/Snapshots'  /snap /Downloads '/.*' )

Then insert --exclude and --include in front of each item in these lists, creating new lists:

exclude_opts=()
for item in "${EXCLUDE[@]}"; do
    exclude_opts+=( --exclude="$item" )
done

include_opts=()
for item in "${INCLUDE[@]}"; do
    include_opts+=( --include="$item" )
done

Then simply use these new lists in your call to rsync (assuming that SOURCE_DIRS is some array of source directories):

rsync ${DRYRUNARG:+"$DRYRUNARG"} -avzhP --delete \
    "${include_opts[@]}" \
    "${exclude_opts[@]}" \
    "${SOURCE_DIRS[@]}" \
    "$MNTPNT/$DEST_DIR"

Here, $DRYRUNARG will be used quoted if it's set and not empty, otherwise it will not be part of the command line at all.

Answer 2

One way would be to use the variable as a file input for --exclude-from for example:

EXCLUDE_DIRS="VirtualBox_VMs
Downloads
Snapshots"

rsync  -arv --exclude-from=<(printf "%s" "$EXCLUDE_DIRS";)  $SOURCE $DESTINATION

Answer 3

I've been trying the same. And find the issue baffling.

The best I could do was (using /bin/bash) turn the comma separate list into list of separate --exclude:

INCLUDE="/.vim*,/.git*,/.ssh,.bash_aliases,.bashrc"
EPATHS=("/VirtualBox_VMs/*/Snapshots" "/snap" "/Downloads" "/.*")     

for path in "${EPATHS[@]}"; do
   EXCLUDE="$EXCLUDE --exclude=$path"
done

rsync -avzhP --delete ${SOURCE_DIRS} \                                                 
    --include=${INCLUDE} \                                                                          
    $EXCLUDE             \                                  
    ${MNTPNT} ${DEST_DIR}

This is an incomplete and inelegant solution, but it works.

The lingering issue is that I cannot get it to work with whitespaces in pathnames. When I tried to add quotes to the concatenation, e.g.

EXCLUDE="$EXCLUDE --exclude=\"$path\""

bash seemed to gum it up with more single-quotes.

As noted before, set -x was my friend here.