Thought I would write a simple script for synchronizing some data, but it turned out to be more difficult than I thought.
The basic layout is that there is a configuration folder, which has subfolders referring to folders that need to be synchronized, and each folder contains [0..2] files (includes.txt & excludes.txt). Then the script would read those through, and run a synchronization command.
What I'd want to run is:
me@my_machine:~/scripts$ aws s3 sync /home/me/Pictures s3://my_bucket/home/me/Pictures --exclude "*" --include "*.gif" --include "*.jpg" --profile=personal --dryrun
(dryrun) upload: ../Pictures/sample_picture.jpg to s3://my_bucket/home/me/Pictures/sample_picture.jpg
So, that I can ignore certain files. I haven't been able to get the excludes and includes to work from a script, as the AWS CLI requires the patterns to be double quoted.
The other questions that I read instructed to use arrays and functions, thus here's my script:
#!/bin/bash
set -x
DRYRUN=true
s3_bucket_uri='s3://my_bucket'
aws_profile='--profile=personal'
config_folder='../config/*'
include_file='includes.txt'
exclude_file='excludes.txt'
includes=()
excludes=()
sync () {
local params=()
local local_folder="$HOME/$1"
local bucket_folder="$s3_bucket_uri""$local_folder"
params+=("$local_folder" "$bucket_folder")
if [[ ${excludes[@]} ]]; then
params+=("${excludes[@]/#/--exclude }")
fi
if [[ ${includes[@]} ]]; then
params+=("${includes[@]/#/--include }")
fi
params+=("$aws_profile")
if [[ "$DRYRUN" = true ]]; then
params+=(--dryrun)
fi
aws s3 sync ${params[@]}
}
read_parameters () {
if [[ -f "$1" ]]; then
while read line; do
if [[ $2 == "include" ]]; then
includes+=("$line")
elif [[ $2 == "exclude" ]]; then
excludes+=("$line")
fi
done < $1
fi
}
reset () {
includes=()
excludes=()
}
for folder in $config_folder; do
if [[ -d "$folder" && ! -L "$folder" ]]; then
read_parameters $folder/$exclude_file exclude
read_parameters $folder/$include_file include
sync "${folder##*/}"
reset
fi
done
With an input example of:
"*.jpg"
"*.gif"
in the includes.txt file.
The problem is getting the quotes correctly for AWS CLI, as it needs double quotes for the inclusion and exclusion patterns, which seem to be quite tricky to get right.
With aws s3 sync ${params[@]}
, the shell adds extra single quotes around the patterns, which doesn't cause the command to crash, but it simply ignores all the patterns:
+ aws s3 sync /home/me/Pictures s3://mybucket/home/me/Pictures --exclude '"*"' --include '"*.gif"' --include '"*.jpg"' --profile=personal --dryrun
(dryrun) upload: ../../../Pictures/Bender_Rodriguez.png to s3://mybucket/home/me/Pictures/Bender_Rodriguez.png
As we can see, it's trying to upload something that should be excluded, as I'm trying to tell it to exclude everything but .gif & .jpg files.
With aws s3 sync "${params[@]}"
the shell adds single quotes around the entire inclusion or exclusion statement, crashing the command:
+ aws s3 sync /home/me/Pictures s3://mybucket/home/me/Pictures '--exclude "*"' '--include "*.gif"' '--include "*.jpg"' --profile=personal --dryrun
Unknown options: --exclude "*",--include "*.gif",--include "*.jpg"
Also tried simply adding a manually created value params+=(--testing "foobar")
, as that was given as the way to go in another question. But that loses all the quotes, and ends up with:
+ aws s3 sync /home/me/Pictures s3://mybucket/home/me/Pictures --testing foobar --profile=personal --dryrun
I did check this question, but even with its answer I get:
bar=( --bar a="b" )
cmd=(foo "${bar[@]}" )
printf '%q ' "${cmd[@]}" && echo # print code equivalent to the command we're about to run
"${cmd[@]}" # actually run this code
+ bar=(--bar a="b")
+ cmd=(foo "${bar[@]}")
+ printf '%q ' foo --bar a=b
foo --bar a=b + echo
+ foo --bar a=b
So, it's losing the double quotes.
Here's my Bash version, in case that makes a difference:
me@my_machine:~/scripts$ bash --version
GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Thus, is there a way to fix this, or should I just rewrite the script in a programming language & use the AWS SDK instead of playing around with shell scripts and AWS CLI?
@muru: If I don't put any quotes in, the inclusion and exclusion patterns are not used:
me@my_machine:~/scripts$ aws s3 sync /home/me/Pictures s3://my_bucket/home/me/Pictures --exclude * --include *.gif --include *.jpg --profile=personal --dryrun
(dryrun) upload: ../Pictures/Bender_Rodriguez.png to s3://my_bucket/home/me/Pictures/Bender_Rodriguez.png
(dryrun) upload: ../Pictures/Panttaus/sormus_paalta.png to s3://my_bucket/home/me/Pictures/Panttaus/sormus_paalta.png
(dryrun) upload: ../Pictures/Panttaus/sormus_sivusta.png to s3://my_bucket/home/me/Pictures/Panttaus/sormus_sivusta.png
(dryrun) upload: ../Pictures/Screenshot from 2021-03-13 22-30-26.png to s3://my_bucket/home/me/Pictures/Screenshot from 2021-03-13 22-30-26.png
(dryrun) upload: ../Pictures/willow_7_months.jpg to s3://my_bucket/home/me/Pictures/willow_7_months.jpg
Same thing happens, if the double quotes are inside single quotes, i.e., what set -x
input shows, if I run:
me@my_machine:~/scripts$ aws s3 sync /home/me/Pictures s3://my_bucket/home/me/Pictures --exclude '"*"' --include '"*.gif"' --include '"*.jpg"' --profile=personal --dryrun
(dryrun) upload: ../Pictures/Bender_Rodriguez.png to s3://my_bucket/home/me/Pictures/Bender_Rodriguez.png
(dryrun) upload: ../Pictures/Panttaus/sormus_paalta.png to s3://my_bucket/home/me/Pictures/Panttaus/sormus_paalta.png
(dryrun) upload: ../Pictures/Panttaus/sormus_sivusta.png to s3://my_bucket/home/me/Pictures/Panttaus/sormus_sivusta.png
(dryrun) upload: ../Pictures/Screenshot from 2021-03-13 22-30-26.png to s3://my_bucket/home/me/Pictures/Screenshot from 2021-03-13 22-30-26.png
(dryrun) upload: ../Pictures/willow_7_months.jpg to s3://my_bucket/home/me/Pictures/willow_7_months.jpg
Only if the double quotes are properly preserved, the exclusion & inclusion patterns work, as mentioned above in the question.
If I remove the quotes completely from the input:
.jpg
.gif
and don't try to add anything in the script either:
aws s3 sync ${params[@]}
The result is single quotes:
+ aws s3 sync /home/me/Pictures s3://my_bucket/home/me/Pictures --exclude '*' --include '*.gif' --include '*.jpg' --profile=personal --dryrun
(dryrun) upload: ../Pictures/Bender_Rodriguez.png to s3://my_bucket/home/me/Pictures/Bender_Rodriguez.png
(dryrun) upload: ../Pictures/Panttaus/sormus_paalta.png to s3://my_bucket/home/me/Pictures/Panttaus/sormus_paalta.png
(dryrun) upload: ../Pictures/Panttaus/sormus_sivusta.png to s3://my_bucket/home/me/Pictures/Panttaus/sormus_sivusta.png
(dryrun) upload: ../Pictures/Screenshot from 2021-03-13 22-30-26.png to s3://my_bucket/home/me/Pictures/Screenshot from 2021-03-13 22-30-26.png
(dryrun) upload: ../Pictures/willow_7_months.jpg to s3://my_bucket/home/me/Pictures/willow_7_months.jpg
Once again, the .png files are not getting ignored.
And with quotes in the script:
aws s3 sync "${params[@]}"
It quotes the whole parameters:
+ aws s3 sync /home/me/Pictures s3://my_bucket/home/me/Pictures '--exclude *' '--include *.gif' '--include *.jpg' --profile=personal --dryrun
Unknown options: --exclude sync.sh,--include *.png,--include *.jpg
Also, just simplifying the script, i.e.:
#!/bin/bash
set -x
DRYRUN=true
s3_bucket_uri='s3://my_bucket'
aws_profile='--profile=personal'
backup_config_folder='../config/*'
include_file='includes.txt'
exclude_file='excludes.txt'
includes=()
excludes=()
sync () {
local params=()
local local_folder="$HOME/$1"
local bucket_folder="$s3_bucket_uri""$local_folder"
params+=("$local_folder" "$bucket_folder")
if [[ ${excludes[@]} ]]; then
params+=("${excludes[@]}")
fi
if [[ ${includes[@]} ]]; then
params+=("${includes[@]}")
fi
params+=("$aws_profile")
if [[ "$DRYRUN" = true ]]; then
params+=(--dryrun)
fi
aws s3 sync "${params[@]}"
}
read_parameters () {
if [[ -f "$1" ]]; then
while read line; do
if [[ $2 == "include" ]]; then
includes+=(--include "$line")
elif [[ $2 == "exclude" ]]; then
excludes+=(--exclude "$line")
fi
done < $1
fi
}
reset () {
includes=()
excludes=()
}
for folder in $backup_config_folder; do
if [[ -d "$folder" && ! -L "$folder" ]]; then
read_parameters $folder/$exclude_file exclude
read_parameters $folder/$include_file include
sync "${folder##*/}"
reset
fi
done
Gives single quotes in the output, and it finally is working.
+ aws s3 sync /home/me/Pictures s3://my_bucket/home/me/Pictures --exclude '*' --include '*.gif' --include '*.jpg' --profile=personal --dryrun
(dryrun) upload: ../../../Pictures/willow_7_months.jpg to s3://my_bucket/home/me/Pictures/willow_7_months.jpg
Thus, I guess the lesson is: don't even try to get double quotes in the first place.
printf
to approximate something like what you want in shell, but IMO it's not worth it. You'll spend all your time and "brain-space" faffing about dealing with whitespace and quoting rather than just using data and variables (scalars and arrays and hashes and dicts etc) where and when you need them.