Behaviour of bash command substitution with command from string in variable

Question

$ echo $(echo x; echo y)
x y
$ a='echo x; echo y'
$ echo $($a)  # expect 'x y'
x; echo y

1. Why does command substitution behave in such way?
2. How to perform the command substitution for list of commands stored in a variable without using eval and bash -c?

echo $(eval $a) and echo $(bash -c "$a") actually does what I want, but I heard that using eval often is a wrong way to solve problems, so I want know how to manage without these commands (using bash -c is in fact just the same thing)

Answer 1

Word splitting happens quite late in the evaluation of a command. Most crucially for you, it happens after variable expansion and command substitution.

This means that the second line in

s="echo a; echo b"
echo $($s)

will first have $s expanded to echo a; echo b and then have that command executed without splitting it into a compound command consisting of two echos.

(Details: $s gets split into four words, echo, a;, echo and b. The encapsulating $(...) executes this as one command, echo with three arguments, a;, echo and b, i.e., you essentially have echo $('echo' 'a;' 'echo' 'b')).

What is given to the outmost echo is, therefore, the string a; echo b (actually three words as $(...) is not in quotes) since this was what was outputted by the innermost echo.

Compare that to

s1="echo a"
s2="echo b"
echo $($s1;$s2)

which results in what you'd expect.

Yes, "eval is evil", most of the time, and sh -c could feel clunky and could be as fragile if you don't know what you're doing. But suppose you have a bit of shell-code that you trust in a string in a shell script. In that case, these tools are the only (?) way to get that code to execute correctly since this often requires explicitly evaluating the text in the string as shell-code (with all evaluation phases from start to finish), especially if it's a compound command.

---

I think it's only due to the Unix shell's intimate relation to text that you're lucky enough to have echo $($s) execute something at all.

Just think about the steps you'd have to take to get a C program to execute a piece of C code that gets given as a string...

Answer 2

After some reading I try to answer by myself

Why does command substitution behave in such way?

$ a='echo x; echo y'
$ echo $($a)  # expect 'x y'

Command substitution

Lets notice that the substitution of variable $a is performed in the subshell produced during command substitution, not in the current shell (1,2,3). Hence the subshell executes the command $a, not echo x; echo y (value of variable a is inherited)

So now we only need to find out why does shell behave like this:

$ a='echo x; echo y'
$ $a
x; echo y

Word splitting

According to Bash Reference Manual:

There are seven kinds of expansion … The order of expansions is: brace expansion; tilde expansion, parameter and variable expansion, arithmetic expansion, and command substitution (done in a left-to-right fashion); word splitting; and filename expansion

— from the section 3.5 Shell Expansions

---

The shell scans the results of parameter expansion, command substitution, and arithmetic expansion that did not occur within double quotes for word splitting.

The shell treats each character of $IFS as a delimiter, and splits the results of the other expansions into words using these characters as field terminators

— from the section 3.5.7 Word Splitting

(The default value of IFS is <space><tab><newline>)

---

The IFS variable is used to split only the results of expansion, not all words (see Word Splitting)

— from the Appendix B Major Differences From The Bourne Shell

---

metacharacter

A character that, when unquoted, separates words. A metacharacter is a space, tab, newline, or one of the following characters: |, &, ;, (, ), <, or >.

— from the section 2. Definitions

---

The brief description of the shell’s operation when it reads and executes a command. Basically, the shell does the following:

1. Reads its input.
2. Breaks the input into words and operators, obeying the quoting rules. These tokens are separated by metacharacters.
3. Parses the tokens into simple and compound commands.
4. Performs the various shell expansions.
5. Performs any necessary redirections.
6. Executes the command

— from the section 3.1.1 Shell Operation

Now we see that there actually are two different kinds of "word splitting" — the initial word splitting (step 2) and word splitting which is kind of shell expansions (step 4).

The initial word splitting (step 2) treats metacharacters like ;, (, ) as delimiters; its outcome is parsed (step 3), so metacharacters and keywords are recognized.

Subsequently bash performs the various shell expansions (step 4) and, among others, "word splitting" — a kind of them. This type of word splitting treats only characters of IFS as delimiters. It does not care about metacharacters. Its outcome is not parsed again, as well as outcome of any other expansion. Thus metacharecters and keywords are not recognized.

That's why ; within value of variable a is not treated as command separator. $a becomes 'echo' 'x;' 'echo' 'y'. Even if value of a is 'echo x ; echo y' the word ';' won't treated as metacharacter, so command $a will become 'echo' 'x' ';' 'echo' 'y'.

Summary:

Metacharacters and keywords (if, then, while etc.) cannot be a result of expansions, but program names, builtin commands, functions and aliases do can. It may introduce some confusion, because it allows to store simple commands in variables, while not allowing to do same thing with compound commands.

$ 'echo' 'a' ';' 'echo' 'b'  # ';' is a literal
a ; echo b

$ cmd=echo
$ "$cmd" a ; $(printf echo) b  # ';' is a metacharacter
a
b

$ cmd='echo a'  # simple command is executed properly
$ $cmd
a

$ cmd='echo a;'  # but metacharacters are not recognized
$ $cmd echo b
a; echo b

$ echo a $(echo ';') echo b  # ';' is a literal
a ; echo b

$ $(printf if) true; then echo a; fi  # parsing error
bash: syntax error near unexpected token 'then'

$ $(printf if) true; $(printf then) echo a; $(printf fi)  # looking for command "if"
bash: if: command not found
bash: then: command not found
bash: fi: command not found

How to perform the command substitution for list of commands stored in a variable without using eval and bash -c (since eval is evil)?

Probably, the right answer is "this is evil too" so you don't need to avoid eval :)

Variables hold data. Functions hold code. Don't put code inside variables!..

— from the article I'm trying to put a command in a variable, but the complex cases always fail!

Links

Unix stackexchange:

Bash shell command substitution

Why is a variable visible in a subshell?

Do parentheses really put the command in a subshell?

Other:

Bash Reference Manual

I'm trying to put a command in a variable, but the complex cases always fail!

conversation in debian bug tracking system