2

Thought I would write a simple script for synchronizing some data, but it turned out to be more difficult than I thought.

The basic layout is that there is a configuration folder, which has subfolders referring to folders that need to be synchronized, and each folder contains [0..2] files (includes.txt & excludes.txt). Then the script would read those through, and run a synchronization command.

What I'd want to run is:

me@my_machine:~/scripts$ aws s3 sync /home/me/Pictures s3://my_bucket/home/me/Pictures --exclude "*" --include "*.gif" --include "*.jpg" --profile=personal --dryrun
(dryrun) upload: ../Pictures/sample_picture.jpg to s3://my_bucket/home/me/Pictures/sample_picture.jpg

So, that I can ignore certain files. I haven't been able to get the excludes and includes to work from a script, as the AWS CLI requires the patterns to be double quoted.

The other questions that I read instructed to use arrays and functions, thus here's my script:

#!/bin/bash

set -x

DRYRUN=true

s3_bucket_uri='s3://my_bucket'
aws_profile='--profile=personal'
config_folder='../config/*'
include_file='includes.txt'
exclude_file='excludes.txt'

includes=()
excludes=()

sync () {
    local params=()
    local local_folder="$HOME/$1"
    local bucket_folder="$s3_bucket_uri""$local_folder"

    params+=("$local_folder" "$bucket_folder")

    if [[ ${excludes[@]} ]]; then
        params+=("${excludes[@]/#/--exclude }")
    fi
    
    if [[ ${includes[@]} ]]; then
        params+=("${includes[@]/#/--include }")
    fi
    
    params+=("$aws_profile")

    if [[ "$DRYRUN" = true ]]; then
        params+=(--dryrun)
    fi

    aws s3 sync ${params[@]}
}


read_parameters () {
    if [[ -f "$1" ]]; then
        while read line; do
            if [[ $2 == "include" ]]; then
                includes+=("$line")
            elif [[ $2 == "exclude" ]]; then
                excludes+=("$line")
            fi
        done < $1
    fi
}

reset () {
    includes=()
    excludes=()
}

for folder in $config_folder; do
    if [[ -d "$folder" && ! -L "$folder" ]]; then
        read_parameters $folder/$exclude_file exclude
        read_parameters $folder/$include_file include
        sync "${folder##*/}"
        reset
    fi
done

With an input example of:

"*.jpg"
"*.gif"

in the includes.txt file.

The problem is getting the quotes correctly for AWS CLI, as it needs double quotes for the inclusion and exclusion patterns, which seem to be quite tricky to get right.

With aws s3 sync ${params[@]}, the shell adds extra single quotes around the patterns, which doesn't cause the command to crash, but it simply ignores all the patterns:

+ aws s3 sync /home/me/Pictures s3://mybucket/home/me/Pictures --exclude '"*"' --include '"*.gif"' --include '"*.jpg"' --profile=personal --dryrun
(dryrun) upload: ../../../Pictures/Bender_Rodriguez.png to s3://mybucket/home/me/Pictures/Bender_Rodriguez.png

As we can see, it's trying to upload something that should be excluded, as I'm trying to tell it to exclude everything but .gif & .jpg files.


With aws s3 sync "${params[@]}" the shell adds single quotes around the entire inclusion or exclusion statement, crashing the command:

+ aws s3 sync /home/me/Pictures s3://mybucket/home/me/Pictures '--exclude "*"' '--include "*.gif"' '--include "*.jpg"' --profile=personal --dryrun
Unknown options: --exclude "*",--include "*.gif",--include "*.jpg"

Also tried simply adding a manually created value params+=(--testing "foobar"), as that was given as the way to go in another question. But that loses all the quotes, and ends up with:

+ aws s3 sync /home/me/Pictures s3://mybucket/home/me/Pictures --testing foobar --profile=personal --dryrun

I did check this question, but even with its answer I get:

bar=( --bar a="b" )
cmd=(foo "${bar[@]}" )
printf '%q ' "${cmd[@]}" && echo  # print code equivalent to the command we're about to run
"${cmd[@]}"                       # actually run this code
+ bar=(--bar a="b")
+ cmd=(foo "${bar[@]}")
+ printf '%q ' foo --bar a=b
foo --bar a=b + echo

+ foo --bar a=b

So, it's losing the double quotes.


Here's my Bash version, in case that makes a difference:

me@my_machine:~/scripts$ bash --version
GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Thus, is there a way to fix this, or should I just rewrite the script in a programming language & use the AWS SDK instead of playing around with shell scripts and AWS CLI?


@muru: If I don't put any quotes in, the inclusion and exclusion patterns are not used:

me@my_machine:~/scripts$ aws s3 sync /home/me/Pictures s3://my_bucket/home/me/Pictures --exclude * --include *.gif --include *.jpg --profile=personal --dryrun
(dryrun) upload: ../Pictures/Bender_Rodriguez.png to s3://my_bucket/home/me/Pictures/Bender_Rodriguez.png
(dryrun) upload: ../Pictures/Panttaus/sormus_paalta.png to s3://my_bucket/home/me/Pictures/Panttaus/sormus_paalta.png
(dryrun) upload: ../Pictures/Panttaus/sormus_sivusta.png to s3://my_bucket/home/me/Pictures/Panttaus/sormus_sivusta.png
(dryrun) upload: ../Pictures/Screenshot from 2021-03-13 22-30-26.png to s3://my_bucket/home/me/Pictures/Screenshot from 2021-03-13 22-30-26.png
(dryrun) upload: ../Pictures/willow_7_months.jpg to s3://my_bucket/home/me/Pictures/willow_7_months.jpg

Same thing happens, if the double quotes are inside single quotes, i.e., what set -x input shows, if I run:

me@my_machine:~/scripts$ aws s3 sync /home/me/Pictures s3://my_bucket/home/me/Pictures --exclude '"*"' --include '"*.gif"' --include '"*.jpg"' --profile=personal --dryrun
(dryrun) upload: ../Pictures/Bender_Rodriguez.png to s3://my_bucket/home/me/Pictures/Bender_Rodriguez.png
(dryrun) upload: ../Pictures/Panttaus/sormus_paalta.png to s3://my_bucket/home/me/Pictures/Panttaus/sormus_paalta.png
(dryrun) upload: ../Pictures/Panttaus/sormus_sivusta.png to s3://my_bucket/home/me/Pictures/Panttaus/sormus_sivusta.png
(dryrun) upload: ../Pictures/Screenshot from 2021-03-13 22-30-26.png to s3://my_bucket/home/me/Pictures/Screenshot from 2021-03-13 22-30-26.png
(dryrun) upload: ../Pictures/willow_7_months.jpg to s3://my_bucket/home/me/Pictures/willow_7_months.jpg

Only if the double quotes are properly preserved, the exclusion & inclusion patterns work, as mentioned above in the question.

If I remove the quotes completely from the input:

.jpg
.gif

and don't try to add anything in the script either:

    aws s3 sync ${params[@]}

The result is single quotes:

+ aws s3 sync /home/me/Pictures s3://my_bucket/home/me/Pictures --exclude '*' --include '*.gif' --include '*.jpg' --profile=personal --dryrun
(dryrun) upload: ../Pictures/Bender_Rodriguez.png to s3://my_bucket/home/me/Pictures/Bender_Rodriguez.png
(dryrun) upload: ../Pictures/Panttaus/sormus_paalta.png to s3://my_bucket/home/me/Pictures/Panttaus/sormus_paalta.png
(dryrun) upload: ../Pictures/Panttaus/sormus_sivusta.png to s3://my_bucket/home/me/Pictures/Panttaus/sormus_sivusta.png
(dryrun) upload: ../Pictures/Screenshot from 2021-03-13 22-30-26.png to s3://my_bucket/home/me/Pictures/Screenshot from 2021-03-13 22-30-26.png
(dryrun) upload: ../Pictures/willow_7_months.jpg to s3://my_bucket/home/me/Pictures/willow_7_months.jpg

Once again, the .png files are not getting ignored.

And with quotes in the script:

    aws s3 sync "${params[@]}"

It quotes the whole parameters:

+ aws s3 sync /home/me/Pictures s3://my_bucket/home/me/Pictures '--exclude *' '--include *.gif' '--include *.jpg' --profile=personal --dryrun

Unknown options: --exclude sync.sh,--include *.png,--include *.jpg

Also, just simplifying the script, i.e.:

#!/bin/bash

set -x

DRYRUN=true

s3_bucket_uri='s3://my_bucket'
aws_profile='--profile=personal'
backup_config_folder='../config/*'
include_file='includes.txt'
exclude_file='excludes.txt'

includes=()
excludes=()

sync () {
    local params=()
    local local_folder="$HOME/$1"
    local bucket_folder="$s3_bucket_uri""$local_folder"

    params+=("$local_folder" "$bucket_folder")

    if [[ ${excludes[@]} ]]; then
        params+=("${excludes[@]}")
    fi
    
    if [[ ${includes[@]} ]]; then
        params+=("${includes[@]}")
    fi

    params+=("$aws_profile")

    if [[ "$DRYRUN" = true ]]; then
        params+=(--dryrun)
    fi

    aws s3 sync "${params[@]}"
}

read_parameters () {
    if [[ -f "$1" ]]; then
        while read line; do
            if [[ $2 == "include" ]]; then
                includes+=(--include "$line")
            elif [[ $2 == "exclude" ]]; then
                excludes+=(--exclude "$line")
            fi
        done < $1
    fi
}

reset () {
    includes=()
    excludes=()
}

for folder in $backup_config_folder; do
    if [[ -d "$folder" && ! -L "$folder" ]]; then
        read_parameters $folder/$exclude_file exclude
        read_parameters $folder/$include_file include
        sync "${folder##*/}"
        reset
    fi
done

Gives single quotes in the output, and it finally is working.

+ aws s3 sync /home/me/Pictures s3://my_bucket/home/me/Pictures --exclude '*' --include '*.gif' --include '*.jpg' --profile=personal --dryrun
(dryrun) upload: ../../../Pictures/willow_7_months.jpg to s3://my_bucket/home/me/Pictures/willow_7_months.jpg

Thus, I guess the lesson is: don't even try to get double quotes in the first place.

10
  • 1
    You're just making this harder for yourself by trying to do it in shell (because you're always going to have issues dealing with quotes and whitespace in any non-trivial shell script). Use perl or python instead - they both have good libraries with AWS & S3 support. e.g. boto3 for python or Amazon::S3 for perl.
    – cas
    Jul 18 at 4:04
  • @cas Very much what I ended up thinking myself too, but thought I'd ask around before giving up.
    – t0mppa
    Jul 18 at 4:11
  • You could probably use printf to approximate something like what you want in shell, but IMO it's not worth it. You'll spend all your time and "brain-space" faffing about dealing with whitespace and quoting rather than just using data and variables (scalars and arrays and hashes and dicts etc) where and when you need them.
    – cas
    Jul 18 at 4:18
  • 1
    Can you show us how a correctly running aws command when typed on the commandline looks like.?
    – guest_7
    Jul 18 at 4:28
  • @guest_7 Added desired outcome & a bit more explanation around it.
    – t0mppa
    Jul 18 at 4:40
4

You're misunderstanding the debug output from set -x. Bash, when logging the commands run due to set -x, shows a canonical-ish quoting, from which you can derive the actual command used by applying quote removal.

Say you had a command like foo "a b" - an argument with spaces in it. When bash needs to log this for set -x, it needs a way to show that a b is a single argument. So, it shows the quoted version in the output - a version that, had it been used in the command line, would have got you this output:

$ foo a\ b
+ foo 'a b'

So say you had to run a command foo, with argument a"b", i.e., the command should receive those quotes, then in bash, you'd typically run some variation of one of these:

foo 'a"b"'
foo a\"b\"
foo 'a"b'\"

Now, when this command needs to be logged, bash must now show the quotes are also quoted, or otherwise one might think that these quotes are due to the problem in the first code block. So, we get:

$ foo a\"b\"
+ foo 'a"b"'

Bash is not adding or removing anything - it's just showing you what it will run, with quoting for clarification.

So, if you see '"*"' in the debug output, you're not seeing bash adding single quotes. You're seeing bash trying to show you that it got "*" after quote removal, so it will have to pass the double quotes on to the command. You should be wondering where those double quotes came from, and I'd guess that's from your input file.


These two code blocks are just unnecessarily complicated:

if [[ ${excludes[@]} ]]; then
    params+=("${excludes[@]/#/--exclude }")
fi

if [[ ${includes[@]} ]]; then
    params+=("${includes[@]/#/--include }")
fi   
if [[ $2 == "include" ]]; then
    includes+=($line)
elif [[ $2 == "exclude" ]]; then
    excludes+=($line)
fi

Just add the option in the when you're reading in the pattern:

if [[ $2 = include ]]; then
    includes+=(--include "$line")
elif [[ $2 = exclude ]]; then
    excludes+=(--exclude "$line")
fi

Then you can just use these arrays directly without further manipulation.

And of course remember to use quotes around variables, unless you want them to undergo field splitting, filename expansion, etc.

12
  • Except that the command doesn't actually work (excludes & includes are ignored) when run via the script, presumably because of quoting. When running from command line with simply the double quotes, it does work. So the bash script is doing something different.
    – t0mppa
    Jul 18 at 4:32
  • @t0mppa "When running from command line with simply the double quotes, it does work." Obviously, yes. Except now (presumably, since you haven't shown us the input files) you're putting those quotes in a file, reading them into a variable, and then using that variable containing quotes in a command. So, no, you are doing something different, not bash.
    – muru
    Jul 18 at 4:38
  • Yes, but if I don't use quotes, the CLI won't pick it up. Thus, I'm trying to figure out how to quote those parts properly in the script. Perhaps I should add quotes in the script then instead of adding them in the input files?
    – t0mppa
    Jul 18 at 4:41
  • "if I don't use quotes, the CLI won't pick it up." You haven't shown us this yet.
    – muru
    Jul 18 at 4:42
  • Edited the question some more to show that.
    – t0mppa
    Jul 18 at 4:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.