0

I have this issue with borgbackup, but because the reaction is the same, I will use rsync in my example.

I want to build an array of arguments by adding a prefix to each, and then give that array to rsync. But rsync acts like it doesn't exist.

With this script:

#!/usr/bin/env bash
#

declare -a exclude_String
for excludestr in $(cat ./list); do
    exclude_String+=(--exclude=$excludestr)
done
rsync "${exclude_String[@]}" . $Destination

and ./list:

'/home/*'
'*.vim*'

A ps of the process running does shows the argument correctly:

/usr/bin/rsync --exclude='*.vim*' --exclude='/home/*' . DESTINATION

But rsync still acts like they aren't there. I found this question on the subject, so I tried:

#!/usr/bin/env bash
#

declare -a exclude_String
exclude_String+=(--exclude='/home/*')
exclude_String+=(--exclude='*.vim*')

rsync "${exclude_String[@]}" . $Destination

And this actually worked.

I don't understand much yet about shell expansion and stuff, I've tried to quote and double quote all over the place just to see if I could find something, but no luck.

Any idea on how to achieve that?

2

If the file actually contains '/home/*', with the quotes, the quotes will be there in the value of excludestr in the loop, and then in the argument you give to rsync. They're not part of a shell command in that case, just part of the expanded value of the command substitution.

On the other hand, if you write exclude_String+=(--exclude='/home/*'), now the quotes are part of a shell command, and are removed during the processing of that command. (With the effect of making * an ordinary character.)

rsync doesn't actually expect or process quotes as part of the exclusion pattern, it will just treat them as any other character, and would exclude files with names containing quotes. So, if you have the exclusion patterns as data in some file, you shouldn't have the quotes there.

Also, instead of for x in $(cat file), use while IFS= read -r line; do ...; done < file, or in this case, mapfile -t exclude_String < file. That command substitution with cat invokes word splitting, so you can't have exclusion patterns containing whitespace. It also invokes globbing of the patterns immediately, right there, on the list of words for the for loop. This isn't what you want.


In general, what you need is:

excludes=()
while IFS= read -r line; do 
    excludes+=(--exclude="$line")
done < ./list 
rsync "$flags" "${excludes[@]}" . "$destination"

With ./list containing just the patterns to exclude:

/home/*
*.vim*

(The quotes around "$line" in +=(--exclude="$line") prevent the shell from globbing files called --exclude=something in the current directory.)

Though in the case of rsync, you might as well use --exclude-from:

rsync "$flags" --exclude-from=./list . "$destination"

If you're going to use something more complicated than a file to feed the loop, you can use process substitution:

while IFS= read -r line; do
    excludes+=(--exclude="$line")
done < <(some command that produces the exclude patterns)

See also: Why is my variable local in one 'while read' loop, but not in another seemingly similar loop?

  • The issue with not putting ' in my source, is that now it is expended before being given to rsync, a ps of the process look like /usr/bin/rsync --exclude=*.vim* --exclude=/home/user1 --exclude=/home/user2 . DESTINATION and hidden file who aren't expended go through the filter. – Vlycop Doo Aug 1 '18 at 17:50
  • @VlycopDoo, yeah... if you use for x in $(cat file), the words in file will be used as globs right there, since the command substitution is unquoted. Another reason to use while read or mapfile, then. I added an example. – ilkkachu Aug 1 '18 at 19:00
  • Strictly speaking, you could use set -f; IFS=$'\n'; for line in $(cat file); do ...; done and you'd get to read the file line-by-line, but I'm not sure that's very nice either. – ilkkachu Aug 1 '18 at 19:03
  • Arggg... i used the cat method to simplify my question, but the way i do it in my real script is way more complicated. This mean that i have to rethink it all in order to never expend the parameter by accident, and without '. Thank's for putting the finger on the issue, but i fear i'm nowhere near done with this ^^ – Vlycop Doo Aug 1 '18 at 19:18
  • @VlycopDoo, it doesn't sound that bad, or at least not worse than shell programming usually. Just quote all the variables ;) (see When is double-quoting necessary? ) If you have a command that produces the exclude patterns, you'll want to check that "Why is my variable local..." question. But the easiest solution is to just use process substitution, which Kusalananda already commented on under their answer. See edit, too. – ilkkachu Aug 1 '18 at 19:27
0

As ilkkachu explained, your issue is that the single quotes should not be part of the patterns and that if you write the patterns in a shell context, the shell will remove them.

However, it looks like you can bypass having to read you patterns explicitly from your file:

rsync --exclude-from=./list source/ target

With --exclude-from=FILENAME, rsync will read the exclusion patterns from the named FILENAME. Note that when doing this, the patterns should not have those single quotes in them.

  • Thank's for your input. i was aware of this solution but it does not fit my use-case because i use a single file who both give what to backup, what to exclude, and what script to run before and after. You could not have know since i oversimplify my script for the question. – Vlycop Doo Aug 1 '18 at 18:03
  • @VlycopDoo Ah, ok. If you're using bash, then you could do rsync --exclude-from=<( command-that-parses-the-file ) .... – Kusalananda Aug 1 '18 at 18:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.