Why do options in a quoted variable fail, but work when unquoted?

Question

I read about that I should quote variables in bash, e.g. "$foo" instead of $foo. However, while writing a script, I found an a case where it works without quotes but not with them:

wget_options='--mirror --no-host-directories'
local_root="$1" # ./testdir recieved from command line
remote_root="$2" # ftp://XXX recieved from command line 
relative_path="$3" # /XXX received from command line

This one works:

wget $wget_options --directory_prefix="$local_root" "$remote_root$relative_path"

This one does not (note the double quotes aroung $wget_options):

wget "$wget_options" --directory_prefix="$local_root" "$remote_root$relative_path"

• What is the reason for this?

• Is the first line the good version; or should I suspect that there is a hidden error somewhere that causes this behavior?

• In general, where do I find good documentation to understand how bash and its quoting works? During writing this script I feel that I started to work on a trial-and-error base instead of understanding the rules.

Answer 1

The most robust way to code that is to use an array:

wget_options=(
    --mirror 
    --no-host-directories
    --directory_prefix="$1"
)
wget "${wget_options[@]}" "$2/$3"

Answer 2

Basically, you should double quote variable expansions to protect them from word splitting (and filename generation). However, in your example,

wget_options='--mirror --no-host-directories'
wget $wget_options --directory_prefix="$local_root" "$remote_root$relative_path"

word splitting is exactly what you want.

With "$wget_options" (quoted), wget doesn't know what to do with the single argument --mirror --no-host-directories and complains

wget: unknown option -- mirror --no-host-directories

For wget to see the two options --mirror and --no-host-directories as separate, word splitting has to occur.

There are more robust ways of doing this. If you are using bash or any other shell that uses arrays like bash do, see glenn jackman's answer. Gilles' answer additionally describes an alternative solution for plainer shells such as the standard /bin/sh. Both essentially store each option as a separate element in an array.

Related question with good answers: Why does my shell script choke on whitespace or other special characters?

---

Double quoting variable expansions is a good rule of thumb. Do that. Then be aware of the very few cases where you shouldn't do that. These will present themselves to you through diagnostic messages, such as the above error message.

There are also a few cases where you don't need to quote variable expansions. But it's easier to continue using double quotes anyway as it doesn't make much difference. One such case is

variable=$other_variable

Another one is

case $variable in
    ...) ... ;;
esac

Answer 3

You're trying to store a list of strings in a string variable. It doesn't fit. No matter how you access the variable, something is broken.

wget_options='--mirror --no-host-directories' sets the variable wget_options to a string that contains a space. At this point, there is no way to know whether the space is supposed to be part of an option, or a separator between options.

When you access the variable with a quoted substitution wget "$wget_options", the value of the variable is used as a string. This means that it's passed as a single parameter to wget, so it's a single option. This breaks in your case because you intended it to mean multiple options.

When you use an unquoted substitution wget $wget_options, the value of the string variable undergoes an expansion process nicknamed “split+glob”:

1. Take the value of the variable and split it into whitespace-delimited parts (assuming you have not modified the $IFS variable). This results in an intermediate list of strings.
2. For each element of the intermediate list, if it is a wildcard pattern that matches one or more files, replace that element by the list of matching files.

This happens to work in your example, because the splitting process turns the space into a separator, but doesn't work in general since an option could contain spaces and wildcard characters.

In ksh, bash, yash and zsh, you can use an array variable. An array in shell terminology is a list of strings, so there is no loss of information. To make an array variable, put parentheses around the array elements when assigning the value to the variable. To access all the elements of the array, use "${VARIABLE[@]}" — this is a generalization of "$@", which forms a list from the elements of the array. Note that you need the double quotes here too, otherwise each element undergoes split+glob.

wget_options=(--mirror --no-host-directories --user-agent="I can haz spaces")
wget "${wget_options[@]}" …

In plain sh, there are no array variables. If you don't mind losing the positional arguments, you can use them to store one list of strings.

set -- --mirror --no-host-directories --user-agent="I can haz spaces"
wget "$@" …

For more information, see Why does my shell script choke on whitespace or other special characters?