Preserving double quotes in a command created from variables in shell script

Question

Thought I would write a simple script for synchronizing some data, but it turned out to be more difficult than I thought.

The basic layout is that there is a configuration folder, which has subfolders referring to folders that need to be synchronized, and each folder contains [0..2] files (includes.txt & excludes.txt). Then the script would read those through, and run a synchronization command.

What I'd want to run is:

me@my_machine:~/scripts$ aws s3 sync /home/me/Pictures s3://my_bucket/home/me/Pictures --exclude "*" --include "*.gif" --include "*.jpg" --profile=personal --dryrun
(dryrun) upload: ../Pictures/sample_picture.jpg to s3://my_bucket/home/me/Pictures/sample_picture.jpg

So, that I can ignore certain files. I haven't been able to get the excludes and includes to work from a script, as the AWS CLI requires the patterns to be double quoted.

The other questions that I read instructed to use arrays and functions, thus here's my script:

#!/bin/bash

set -x

DRYRUN=true

s3_bucket_uri='s3://my_bucket'
aws_profile='--profile=personal'
config_folder='../config/*'
include_file='includes.txt'
exclude_file='excludes.txt'

includes=()
excludes=()

sync () {
    local params=()
    local local_folder="$HOME/$1"
    local bucket_folder="$s3_bucket_uri""$local_folder"

    params+=("$local_folder" "$bucket_folder")

    if [[ ${excludes[@]} ]]; then
        params+=("${excludes[@]/#/--exclude }")
    fi
    
    if [[ ${includes[@]} ]]; then
        params+=("${includes[@]/#/--include }")
    fi
    
    params+=("$aws_profile")

    if [[ "$DRYRUN" = true ]]; then
        params+=(--dryrun)
    fi

    aws s3 sync ${params[@]}
}


read_parameters () {
    if [[ -f "$1" ]]; then
        while read line; do
            if [[ $2 == "include" ]]; then
                includes+=("$line")
            elif [[ $2 == "exclude" ]]; then
                excludes+=("$line")
            fi
        done < $1
    fi
}

reset () {
    includes=()
    excludes=()
}

for folder in $config_folder; do
    if [[ -d "$folder" && ! -L "$folder" ]]; then
        read_parameters $folder/$exclude_file exclude
        read_parameters $folder/$include_file include
        sync "${folder##*/}"
        reset
    fi
done

With an input example of:

"*.jpg"
"*.gif"

in the includes.txt file.

The problem is getting the quotes correctly for AWS CLI, as it needs double quotes for the inclusion and exclusion patterns, which seem to be quite tricky to get right.

With aws s3 sync ${params[@]}, the shell adds extra single quotes around the patterns, which doesn't cause the command to crash, but it simply ignores all the patterns:

+ aws s3 sync /home/me/Pictures s3://mybucket/home/me/Pictures --exclude '"*"' --include '"*.gif"' --include '"*.jpg"' --profile=personal --dryrun
(dryrun) upload: ../../../Pictures/Bender_Rodriguez.png to s3://mybucket/home/me/Pictures/Bender_Rodriguez.png

As we can see, it's trying to upload something that should be excluded, as I'm trying to tell it to exclude everything but .gif & .jpg files.

---

With aws s3 sync "${params[@]}" the shell adds single quotes around the entire inclusion or exclusion statement, crashing the command:

+ aws s3 sync /home/me/Pictures s3://mybucket/home/me/Pictures '--exclude "*"' '--include "*.gif"' '--include "*.jpg"' --profile=personal --dryrun
Unknown options: --exclude "*",--include "*.gif",--include "*.jpg"

Also tried simply adding a manually created value params+=(--testing "foobar"), as that was given as the way to go in another question. But that loses all the quotes, and ends up with:

+ aws s3 sync /home/me/Pictures s3://mybucket/home/me/Pictures --testing foobar --profile=personal --dryrun

---

I did check this question, but even with its answer I get:

bar=( --bar a="b" )
cmd=(foo "${bar[@]}" )
printf '%q ' "${cmd[@]}" && echo  # print code equivalent to the command we're about to run
"${cmd[@]}"                       # actually run this code

+ bar=(--bar a="b")
+ cmd=(foo "${bar[@]}")
+ printf '%q ' foo --bar a=b
foo --bar a=b + echo

+ foo --bar a=b

So, it's losing the double quotes.

---

Here's my Bash version, in case that makes a difference:

me@my_machine:~/scripts$ bash --version
GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Thus, is there a way to fix this, or should I just rewrite the script in a programming language & use the AWS SDK instead of playing around with shell scripts and AWS CLI?

---

@muru: If I don't put any quotes in, the inclusion and exclusion patterns are not used:

me@my_machine:~/scripts$ aws s3 sync /home/me/Pictures s3://my_bucket/home/me/Pictures --exclude * --include *.gif --include *.jpg --profile=personal --dryrun
(dryrun) upload: ../Pictures/Bender_Rodriguez.png to s3://my_bucket/home/me/Pictures/Bender_Rodriguez.png
(dryrun) upload: ../Pictures/Panttaus/sormus_paalta.png to s3://my_bucket/home/me/Pictures/Panttaus/sormus_paalta.png
(dryrun) upload: ../Pictures/Panttaus/sormus_sivusta.png to s3://my_bucket/home/me/Pictures/Panttaus/sormus_sivusta.png
(dryrun) upload: ../Pictures/Screenshot from 2021-03-13 22-30-26.png to s3://my_bucket/home/me/Pictures/Screenshot from 2021-03-13 22-30-26.png
(dryrun) upload: ../Pictures/willow_7_months.jpg to s3://my_bucket/home/me/Pictures/willow_7_months.jpg

Same thing happens, if the double quotes are inside single quotes, i.e., what set -x input shows, if I run:

me@my_machine:~/scripts$ aws s3 sync /home/me/Pictures s3://my_bucket/home/me/Pictures --exclude '"*"' --include '"*.gif"' --include '"*.jpg"' --profile=personal --dryrun
(dryrun) upload: ../Pictures/Bender_Rodriguez.png to s3://my_bucket/home/me/Pictures/Bender_Rodriguez.png
(dryrun) upload: ../Pictures/Panttaus/sormus_paalta.png to s3://my_bucket/home/me/Pictures/Panttaus/sormus_paalta.png
(dryrun) upload: ../Pictures/Panttaus/sormus_sivusta.png to s3://my_bucket/home/me/Pictures/Panttaus/sormus_sivusta.png
(dryrun) upload: ../Pictures/Screenshot from 2021-03-13 22-30-26.png to s3://my_bucket/home/me/Pictures/Screenshot from 2021-03-13 22-30-26.png
(dryrun) upload: ../Pictures/willow_7_months.jpg to s3://my_bucket/home/me/Pictures/willow_7_months.jpg

Only if the double quotes are properly preserved, the exclusion & inclusion patterns work, as mentioned above in the question.

If I remove the quotes completely from the input:

.jpg
.gif

and don't try to add anything in the script either:

    aws s3 sync ${params[@]}

The result is single quotes:

+ aws s3 sync /home/me/Pictures s3://my_bucket/home/me/Pictures --exclude '*' --include '*.gif' --include '*.jpg' --profile=personal --dryrun
(dryrun) upload: ../Pictures/Bender_Rodriguez.png to s3://my_bucket/home/me/Pictures/Bender_Rodriguez.png
(dryrun) upload: ../Pictures/Panttaus/sormus_paalta.png to s3://my_bucket/home/me/Pictures/Panttaus/sormus_paalta.png
(dryrun) upload: ../Pictures/Panttaus/sormus_sivusta.png to s3://my_bucket/home/me/Pictures/Panttaus/sormus_sivusta.png
(dryrun) upload: ../Pictures/Screenshot from 2021-03-13 22-30-26.png to s3://my_bucket/home/me/Pictures/Screenshot from 2021-03-13 22-30-26.png
(dryrun) upload: ../Pictures/willow_7_months.jpg to s3://my_bucket/home/me/Pictures/willow_7_months.jpg

Once again, the .png files are not getting ignored.

And with quotes in the script:

    aws s3 sync "${params[@]}"

It quotes the whole parameters:

+ aws s3 sync /home/me/Pictures s3://my_bucket/home/me/Pictures '--exclude *' '--include *.gif' '--include *.jpg' --profile=personal --dryrun

Unknown options: --exclude sync.sh,--include *.png,--include *.jpg

Also, just simplifying the script, i.e.:

#!/bin/bash

set -x

DRYRUN=true

s3_bucket_uri='s3://my_bucket'
aws_profile='--profile=personal'
backup_config_folder='../config/*'
include_file='includes.txt'
exclude_file='excludes.txt'

includes=()
excludes=()

sync () {
    local params=()
    local local_folder="$HOME/$1"
    local bucket_folder="$s3_bucket_uri""$local_folder"

    params+=("$local_folder" "$bucket_folder")

    if [[ ${excludes[@]} ]]; then
        params+=("${excludes[@]}")
    fi
    
    if [[ ${includes[@]} ]]; then
        params+=("${includes[@]}")
    fi

    params+=("$aws_profile")

    if [[ "$DRYRUN" = true ]]; then
        params+=(--dryrun)
    fi

    aws s3 sync "${params[@]}"
}

read_parameters () {
    if [[ -f "$1" ]]; then
        while read line; do
            if [[ $2 == "include" ]]; then
                includes+=(--include "$line")
            elif [[ $2 == "exclude" ]]; then
                excludes+=(--exclude "$line")
            fi
        done < $1
    fi
}

reset () {
    includes=()
    excludes=()
}

for folder in $backup_config_folder; do
    if [[ -d "$folder" && ! -L "$folder" ]]; then
        read_parameters $folder/$exclude_file exclude
        read_parameters $folder/$include_file include
        sync "${folder##*/}"
        reset
    fi
done

Gives single quotes in the output, and it finally is working.

+ aws s3 sync /home/me/Pictures s3://my_bucket/home/me/Pictures --exclude '*' --include '*.gif' --include '*.jpg' --profile=personal --dryrun
(dryrun) upload: ../../../Pictures/willow_7_months.jpg to s3://my_bucket/home/me/Pictures/willow_7_months.jpg

Thus, I guess the lesson is: don't even try to get double quotes in the first place.

Answer 1

You're misunderstanding the debug output from set -x. Bash, when logging the commands run due to set -x, shows a canonical-ish quoting, from which you can derive the actual command used by applying quote removal.

Say you had a command like foo "a b" - an argument with spaces in it. When bash needs to log this for set -x, it needs a way to show that a b is a single argument. So, it shows the quoted version in the output - a version that, had it been used in the command line, would have got you this output:

$ foo a\ b
+ foo 'a b'

So say you had to run a command foo, with argument a"b", i.e., the command should receive those quotes, then in bash, you'd typically run some variation of one of these:

foo 'a"b"'
foo a\"b\"
foo 'a"b'\"

Now, when this command needs to be logged, bash must now show the quotes are also quoted, or otherwise one might think that these quotes are due to the problem in the first code block. So, we get:

$ foo a\"b\"
+ foo 'a"b"'

Bash is not adding or removing anything - it's just showing you what it will run, with quoting for clarification.

So, if you see '"*"' in the debug output, you're not seeing bash adding single quotes. You're seeing bash trying to show you that it got "*" after quote removal, so it will have to pass the double quotes on to the command. You should be wondering where those double quotes came from, and I'd guess that's from your input file.

---

These two code blocks are just unnecessarily complicated:

if [[ ${excludes[@]} ]]; then
    params+=("${excludes[@]/#/--exclude }")
fi

if [[ ${includes[@]} ]]; then
    params+=("${includes[@]/#/--include }")
fi   

if [[ $2 == "include" ]]; then
    includes+=($line)
elif [[ $2 == "exclude" ]]; then
    excludes+=($line)
fi

Just add the option in the when you're reading in the pattern:

if [[ $2 = include ]]; then
    includes+=(--include "$line")
elif [[ $2 = exclude ]]; then
    excludes+=(--exclude "$line")
fi

Then you can just use these arrays directly without further manipulation.

And of course remember to use quotes around variables, unless you want them to undergo field splitting, filename expansion, etc.