6

Consider the following Bash script:

args="--var 'lddpro_subdomain=prodcopy-lddpro' --var 'partner_subdomain=prodcopy-partner' --var 'model_design_subdomain=prodcopy-model-design'"
for x in ${args}; do echo ${x}; done

# Output
--var
'lddpro_subdomain=prodcopy-lddpro'
--var
'partner_subdomain=prodcopy-partner'
--var
'model_design_subdomain=prodcopy-model-design'

My problem is that the single quotes aren't removed.

This is a problem, when I want to invoke Terraform with args:

terraform apply ... ${args}

where args is a script parameter passed by the caller, e.g.

deploy.sh --args="--var 'lddpro_subdomain=prodcopy-lddpro' --var 'partner_subdomain=prodcopy-partner' --var 'model_design_subdomain=prodcopy-model-design'"

In this case, Terraform complains that 'lddpro_subdomain doesn't refer to a valid variable declared in the configuration.

In constrast, the above Terraform command would be valid, if Bash had removed the single quotes when performing parameter expansion.

The manpage of Bash 5.2 mentions:

Quote Removal
    After the preceding expansions, all unquoted occurrences of the characters \, ', and " that did not result from one of the above expansions are removed.

How can I make Bash perform quote removal after parameter expansion, or by other means make ${args} expand to --var lddpro_subdomain=prodcopy-lddpro --var partner_subdomain=prodcopy-partner --var model_design_subdomain=prodcopy-model-design", so that Terraform doesn't consider the single quote (') as part of the variable name?

4
  • 4
    Attempting to store arguments in a string variable are problematic. This is a common FAQ. mywiki.wooledge.org/BashFAQ/050
    – tripleee
    Commented Jun 10 at 10:12
  • 1
    And within the SE universe: stackoverflow.com/questions/12136948/…
    – Barmar
    Commented Jun 10 at 15:23
  • On first thought, perhaps just adding an eval to have it process the line one more time. In your example script, just using eval echo ${x} should get that job done. Evaling your call to terraform might be all that's needed.
    – penguin359
    Commented Jun 10 at 16:31
  • Change deploy.sh --args="--var 'lddpro_subdomain=prodcopy-lddpro' --var 'partner_subdomain=prodcopy-partner' --var 'model_design_subdomain=prodcopy-model-design'" to deploy.sh --args="--var lddpro_subdomain=prodcopy-lddpro --var partner_subdomain=prodcopy-partner --var model_design_subdomain=prodcopy-model-design" and your immediate problem is gone. If your real data is more complex you'll need BashFAQ #50 or equivalents, but there's no evidence that that's true in the immediate case. Commented Jun 11 at 15:54

4 Answers 4

18

The way to remove a particular string from a variable is to use the ${var/pattern/replacement} expansion. With a doubled slash, it handles all instances of the string, and with an empty replacement, the slash before replacement can be omitted. So

$ var="ain't so"
$ echo "${var//"'"}"
aint so

However, that won't help you with the fact that word splitting doesn't care about the quotes, so the string will still be split on any whitespace between the quotes. Meaning that the quotes are rather useless to begin with. Not that your example string here even needs to protect whitespace, since you have none within the values, but try e.g. your script with args="--var 'foo bar'" and see what happens. Also remember that glob characters in the string will still try to match filenames after variable expansion and word splitting.

Instead of trying to force multiple distinct strings into one, rather use an array, e.g.

args=(--var 'foo bar')
somecmd "${args[@]}"

or with the arguments you show:

args=(--var 'lddpro_subdomain=prodcopy-lddpro'
      --var 'partner_subdomain=prodcopy-partner'
      --var 'model_design_subdomain=prodcopy-model-design')
terraform apply ... "${args[@]}"

(though you don't need the quotes here anyway, as a-z, _, - and = aren't special enough to the shell.)

See: How can we run a command stored in a variable?

11
  • Thank you, I guess you mean that word splitting doesn't care about the quotes. I can't provide an array, as the script itself takes a parameter --tf-extra-args as a single argument to be provided by e.g. GitHub Actions :)
    – Shuzheng
    Commented Jun 9 at 11:15
  • 1
    @Shuzheng I can't help but wonder, why include them in the first place?
    – Tom Yan
    Commented Jun 9 at 11:18
  • 7
    @Shuzheng, well, you're not showing an argument like --tf-extra-args in the actual post, so we don't know what that is and how it should be taken into account... Instead, what you do show, is an assigment of multiple args to a single string, and the mention that then using the unquoted ${args} doesn't do what you want. That's exactly the case where using an array would be better. Then again, if your original question doesn't represent your actual situation, perhaps you'd like to edit it to show what you actually do have?
    – ilkkachu
    Commented Jun 9 at 15:53
  • 5
    @Shuzheng Why pass all those arguments as a single string to your script? Wouldn't your script be able to take them as a proper list of arguments? The problem begins when you star putting multiple separate strings into a single string, which means you will, at some point, have to parse that single string to extract the multiple strings in their original form, which isn't easy (basically, you would need to reimplement the shell's own command line parsing).
    – Kusalananda
    Commented Jun 9 at 17:50
  • 1
    @Shuzheng, or, if you don't need the quotes, don't put them there. That is, if you know the arguments concatenated in that string are simple enough that there are no spaces in the values themselves, just the ones used to separate the values. Or, if the plan is to pass a string that gets interpreted as shell code, then you need to pass it through eval. (Which is a security concern or not, depending on if there is a security boundary in between.)
    – ilkkachu
    Commented Jun 10 at 7:25
4

You should use an array to store arguments:

args=(--var 'lddpro_subdomain=prodcopy-lddpro' --var 'partner_subdomain=prodcopy-partner' --var 'model_design_subdomain=prodcopy-model-design')

terraform apply "${args[@]}"

The POSIX equivalent would be to use $@ as an array. (This can be tricky if you can't set aside the current contents of $@.)

set -- --var 'lddpro_subdomain=prodcopy-lddpro' --var 'partner_subdomain=prodcopy-partner' --var 'model_design_subdomain=prodcopy-model-design'

terraform apply "$@"
2

To make Bash remove these quotes (and others such as " as well) according to its own argument parsing rules, it's possible (but not secure) to do it wih eval + set:

args="--var 'lddpro_subdomain=prodcopy-lddpro' --var 'partner_subdomain=prodcopy-partner' --var 'model_design_subdomain=prodcopy-model-design'"
eval "set . $args"
shift  # Remove the dot (.) from the front of "$@".
for x in "$@"; do echo "$x"; done

Output:

--var
lddpro_subdomain=prodcopy-lddpro
--var
partner_subdomain=prodcopy-partner
--var
model_design_subdomain=prodcopy-model-design

However, it's insecure, because it's vulnerable to command injection: if the attacker controls the value of "$args", then they can run arbitrary commands, e.g. with args="; rm -irf ~ /", running the eval in the script above will remove files recursively.

In general, avoid eval with arguments supplied by the caller.


The equivalent eval + array-assignment is also vulnerable:

args="--var 'lddpro_subdomain=prodcopy-lddpro' --var 'partner_subdomain=prodcopy-partner' --var 'model_design_subdomain=prodcopy-model-design'"
eval "args2=($args)"
for x in "${args2[@]}"; do echo "$x"; done

Example "$args" assignment the attacker can do to remove files recursively: args="); rm -irf ~ /; args2=(".

3
  • 1
    What are the dot and shift about?
    – Tom Yan
    Commented Jun 10 at 6:00
  • 2
    Note that you can use eval "set -- $args" instead of adding that . and removing it afterwards. Also note that "$@" is what for loops loop over by default, so you can just do for x do printf '%s\n' "$x"; done. Commented Jun 10 at 6:30
  • @TomYan: The dot is there to make it work in case the first argument starts with a - (which set would otherwise interpret as a command-line flag). As demonstrated in the comment by @StéphaneChazelas, it's possible to make it work without a dot.
    – pts
    Commented Jun 10 at 10:34
2

I am going to answer a slightly more general question, namely how to escape and revert a list of arguments. The transform you want is a special case.

Suppose the original array is:

SOME_ARR=(--var lddpro_subdomain=prodcopy-lddpro --var "partner_subdomain=prodcopy partner" --var model_design_subdomain=prodcopy-model-design)

then we can get at your args by parameter expansion

$ args="${SOME_ARR[*]@Q}"
$ echo $args
'--var' 'lddpro_subdomain=prodcopy-lddpro' '--var' 'partner_subdomain=prodcopy partner' '--var' 'model_design_subdomain=prodcopy-model-design'

To revert this you can declare an array back [1]

$ declare -a SOME_ARR_RECOVERED="(${args:-})"
$ declare | grep ^SOME_ARR_RECOVERED
SOME_ARR_RECOVERED=([0]="--var" [1]="lddpro_subdomain=prodcopy-lddpro" [2]="--var" [3]="partner_subdomain=prodcopy partner" [4]="--var" [5]="model_design_subdomain=prodcopy-model-design")

and process it, in this case, to concatenate it into a single string without quotes

$ out=""
$ for i in "${SOME_ARR_RECOVERED[@]}"; do out="$out $i"; echo $i; done
--var
lddpro_subdomain=prodcopy-lddpro
--var
partner_subdomain=prodcopy partner
--var
model_design_subdomain=prodcopy-model-design
$ echo "$out"
--var lddpro_subdomain=prodcopy-lddpro --var partner_subdomain=prodcopy partner --var model_design_subdomain=prodcopy-model-design

which seems to be what you want. It is perhaps an overkill and ${args//"'"/} would work equally well, assuming you can control $args and can ensure that you won't remove any (inner) escaped single quotes, that you don't have the relevant IFS contained within any single array member, etc.


[1] as ilkkachu pointed out in the comments, this is also insecure, e.g.,

$ args='`mktemp`'; declare -a A="($args)"; echo ${A[@]}
/tmp/tmp.seBvzUiN1K
New contributor
teapot is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
2
  • 2
    FWIW, declare here acts essentially like eval, and will run e.g. command substitutions if they come up in $args.
    – ilkkachu
    Commented Jun 10 at 14:13
  • @ilkkachu Good point, added.
    – teapot
    Commented Jun 10 at 14:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .