This has been discussed in a number of questions on unix.SE, I'll try to collect all issues I can come up with here. Below is a description of why and how the various attempts fail, a way to do it properly with a function (for a fixed command), or with shell arrays (Bash/ksh/zsh) or the $@
pseudo-array (POSIX sh), and notes about using eval
to do this. Some references at the end.
For the purposes here, it doesn't matter if it's only the command arguments or also the command name that is to be stored in a variable. They're processed similarly up to the point where the command is launched, at which point the shell just takes the first word as the name of the command to run.
Why it fails
The reason you face those problems is word splitting and the fact that quotes expanded from variables don't act as quotes, but are just ordinary characters.
(Note that this is similar to every other programming language: e.g. char *s = "foo()"; printf("%s\n", s)
does not call the function foo()
in C, but just prints the string foo()
. The shell is a programming language, not a macro processor.)
Remember that its the shell that processes quotes and variable expansions on the command line, turning it from a single string into the list of strings that eventually get passed to the launched command. The program itself doesn't see any quotes. E.g. if given the command ls -l "foo bar"
, the shell turns that into the three strings ls
, -l
and foo bar
(removing the quotes), and passes those to ls
. (Even the command name is passed, though not all programs use it.)
The cases presented in the question:
The assignment here assigns the single string ls -l "/tmp/test/my dir"
to abc
:
$ abc='ls -l "/tmp/test/my dir"'
Below, $abc
is split on whitespace, and ls
gets the three arguments -l
, "/tmp/test/my
and dir"
(with a quote at the front of the second and another at the back of the third). The option works, but the path gets incorrectly processed:
$ $abc
ls: cannot access '"/tmp/test/my': No such file or directory
ls: cannot access 'dir"': No such file or directory
Here, the expansion is quoted, so it's kept as a single word. The shell tries to find a program literally called ls -l "/tmp/test/my dir"
, spaces and quotes included.
$ "$abc"
bash: ls -l "/tmp/test/my dir": No such file or directory
And here, $abc
is split, and only the first resulting word is taken as the argument to -c
, so Bash just runs ls
in the current directory. The other words are arguments to bash, and are used to fill $0
, $1
, etc.
$ bash -c $abc
'my dir'
With bash -c "$abc"
, and eval "$abc"
, there's an additional shell processing step, which does make the quotes work, but also causes all shell expansions to be processed again, so there's a risk of accidentally running e.g. a command substitution from user-provided data, unless you're very careful about quoting.
Better ways to do it
The two better ways to store a command are a) use a function instead, b) use an array variable (or the positional parameters).
Using a function:
Simply declare a function with the command inside, and run the function as if it were a command. Expansions in commands within the function are only processed when the command runs, not when it's defined, and you don't need to quote the individual commands.
# define it
myls() {
ls -l "/tmp/test/my dir"
}
# run it
myls
Using an array:
Arrays allow creating multi-word variables where the individual words contain white space. Here, the individual words are stored as distinct array elements, and the "${array[@]}"
expansion expands each element as separate shell words:
# define the array
mycmd=(ls -l "/tmp/test/my dir")
# run the command
"${mycmd[@]}"
The syntax is slightly horrible, but arrays also allow you to build the command line piece-by-piece. For example:
mycmd=(ls) # initial command
if [ "$want_detail" = 1 ]; then
mycmd+=(-l) # optional flag
fi
mycmd+=("$targetdir") # the filename
"${mycmd[@]}"
or keep parts of the command line constant and use the array fill just a part of it, like options or filenames:
options=(-x -v)
files=(file1 "file name with whitespace")
target=/somedir
transmutate "${options[@]}" "${files[@]}" "$target"
The downside of arrays is that they're not a standard feature, so plain POSIX shells (like dash
, the default /bin/sh
in Debian/Ubuntu) don't support them (but see below). Bash, ksh and zsh do, however, so it's likely your system has some shell that supports arrays.
Using "$@"
In shells with no support for named arrays, one can still use the positional parameters (the pseudo-array "$@"
) to hold the arguments of a command.
The following should be portable script bits that do the equivalent of the code bits in the previous section. The array is replaced with "$@"
, the list of positional parameters. Setting "$@"
is done with set
, and the double quotes around "$@"
are important (these cause the elements of the list to be individually quoted).
First, simply storing a command with arguments in "$@"
and running it:
set -- ls -l "/tmp/test/my dir"
"$@"
Conditionally setting parts of the command line options for a command:
set -- ls
if [ "$want_detail" = 1 ]; then
set -- "$@" -l
fi
set -- "$@" "$targetdir"
"$@"
Only using "$@"
for options and operands:
set -- -x -v
set -- "$@" file1 "file name with whitespace"
set -- "$@" /somedir
transmutate "$@"
(Of course, "$@"
is usually filled with the arguments to the script itself, so you'll have to save them somewhere before re-purposing "$@"
.)
Using eval
(be careful here!)
eval
takes a string and runs it as a command, just like if it was entered on the shell command line. This includes all quote and expansion processing, which is both useful and dangerous.
In the simple case, it allows doing just what we want:
cmd='ls -l "/tmp/test/my dir"'
eval "$cmd"
With eval
, the quotes are processed, so ls
eventually sees just the two arguments -l
and /tmp/test/my dir
, like we want. eval
is also smart enough to concatenate any arguments it gets, so eval $cmd
could also work in some cases, but e.g. all runs of whitespace would be changed to single spaces. It's still better to quote the variable there as that will ensure it gets unmodified to eval
.
However, it's dangerous to include user input in the command string to eval
. For example, this seems to work:
read -r filename
cmd="ls -ld '$filename'"
eval "$cmd";
But if the user gives input that contains single quotes, they can break out of the quoting and run arbitrary commands! E.g. with the input '$(whatever)'.txt
, your script happily runs the command substitution. That it could have been rm -rf
(or worse) instead.
The issue there is that the value of $filename
was embedded in the command line that eval
runs. It was expanded before eval
, which saw e.g. the command ls -l ''$(whatever)'.txt'
. You would need to pre-process the input to be safe.
If we do it the other way, keeping the filename in the variable, and letting the eval
command expand it, it's safer again:
read -r filename
cmd='ls -ld "$filename"'
eval "$cmd";
Note the outer quotes are now single quotes, so expansions within do not happen. Hence, eval
sees the command ls -l "$filename"
and expands the filename safely itself.
But that's not much different from just storing the command in a function or an array. With functions or arrays, there is no such problem since the words are kept separate for the whole time, and there's no quote or other processing for the contents of filename
.
read -r filename
cmd=(ls -ld -- "$filename")
"${cmd[@]}"
Pretty much the only reason to use eval
is one where the varying part involves shell syntax elements that can't be brought in via variables (pipelines, redirections, etc.). However, you'll then need to quote/escape everything else on the command line that needs protection from the additional parsing step (see link below). In any case, it's best to avoid embedding input from the user in the eval
command!
References