1
$ echo $(echo x; echo y)
x y
$ a='echo x; echo y'
$ echo $($a)  # expect 'x y'
x; echo y
  1. Why does command substitution behave in such way?
  2. How to perform the command substitution for list of commands stored in a variable without using eval and bash -c?

echo $(eval $a) and echo $(bash -c "$a") actually does what I want, but I heard that using eval often is a wrong way to solve problems, so I want know how to manage without these commands (using bash -c is in fact just the same thing)

3

Word splitting happens quite late in the evaluation of a command. Most crucially for you, it happens after variable expansion and command substitution.

This means that the second line in

s="echo a; echo b"
echo $($s)

will first have $s expanded to echo a; echo b and then have that command executed without splitting it into a compound command consisting of two echos.

(details: $s gets split into four words, echo, a;, echo and b. The encapsulating $(...) executes this as one command, echo with three arguments, a;, echo and b.)

What is given to the outmost echo is therefore the string a; echo b (actually three words as $(...) is not in quotes) since this was what was outputted by the innermost echo.

Compare that to

s1="echo a"
s2="echo b"
echo $($s1;$s2)

which results in what you'd expect.

Yes, "eval is evil", most of the time, and sh -c is clunky and as fragile. But if you have a bit of shell code that you trust in a string in a shell script, then these tools are the only (?) way to get that code to execute properly since this often requires having to explicitly evaluate the text in the string as shell code (with all phases of the evaluation from start to finish). Especially if it's a compound command.


I think it's only due to the Unix shell's intimate relation to text that you're lucky enough to have echo $($s) execute something at all.

Just think about the steps you'd have to take to get a C program to execute a piece of C code that it gets given as a string...

  • 2
    More exactly, $s is first split into echo, a;, echo and b (assuming a default value of $IFS), and the first word is used to derive the command to execute (echo, in this case a builtin) and all 4 arguments are passed to echo. echo outputs the 2nd to last with space characters in between and a newline at the end. $() strips that newline and there's a second round of split+glob. Which results in a;, echo and b to be passed to the outer echo as well (assuming again a default value of $IFS). – Stéphane Chazelas Aug 25 '17 at 16:25
3

After some reading I will try to answer more particularly.

Why does command substitution behave in such way?

$ a='echo x; echo y'
$ echo $($a)  # expect 'x y'

Command substitution

Lets notice that the substitution of variable $a is performing in the subshell produced during command substitution, not in the current shell (1,2,3). Hence the subshell executes the command $a, not echo x; echo y (value of variable a is inherited)

So now we only need to understand why does shell behave in such way:

$ a='echo x; echo y'
$ $a
x; echo y

Word splitting

According to [Bash Reference Manual]:

… There are seven kinds of expansion … The order of expansions is: brace expansion; tilde expansion, parameter and variable expansion, arithmetic expansion, and command substitution (done in a left-to-right fashion); word splitting; and filename expansion …

— from the section 3.5 Shell Expansions


… The shell scans the results of parameter expansion, command substitution, and arithmetic expansion that did not occur within double quotes for word splitting.

The shell treats each character of $IFS as a delimiter, and splits the results of the other expansions into words using these characters as field terminators …

— from the section 3.5.7 Word Splitting

(The default value of IFS is <space><tab><newline>)


… The IFS variable is used to split only the results of expansion, not all words (see Word Splitting) …

— from the Appendix B Major Differences From The Bourne Shell


metacharacter

A character that, when unquoted, separates words. A metacharacter is a space, tab, newline, or one of the following characters: |, &, ;, (, ), <, or >.

— from the section 2. Definitions


The brief description of the shell’s operation when it reads and executes a command. Basically, the shell does the following:

  1. Reads its input.
  2. Breaks the input into words and operators, obeying the quoting rules. These tokens are separated by metacharacters.
  3. Parses the tokens into simple and compound commands.
  4. Performs the various shell expansions.
  5. Performs any necessary redirections.
  6. Executes the command

— from the section 3.1.1 Shell Operation

So now we can see that there are two different kinds of "word splitting" — the "inital" word splitting (step 2) and word splitting that is kind of shell expansions (step 4).

The "inital" word splitting (step 2) respects metacharacters like ;, (, ). The outcome of "inital" word splitting is parsed (step 3) and all its metacharacters and keywords are recognized.

Subsequently bash performs the various shell expansions (step 4) and, among others, word splitting (as one kind of expansions). "This" word splitting treats only characters of IFS as delimiters. "This" word splitting does not respect metacharacters. Outcome of "this" word splitting is not parsed again as well as outcome of any expansion. Thus metacharecters and keywords in the result of expansions are not recognized.

That's why ; within value of variable a is not treated as metacharacter. $a becomes 'echo' 'x;' 'echo' 'y'. Even if value of a was 'echo x ; echo y' the word ';' would not be treated as metacharacter and command $a would become 'echo' 'x' ';' 'echo' 'y'.

Summary:

Metacharacters and keywords (if, then, while etc.) cannot be a result of expansions. But program names (as well as builtin commands, functions and aliases) can . It is a source of confusion, because it allows to put simple commands in variables and doesn't allow to put compound ones.

$ 'echo' 'a' ';' 'echo' 'b'  # ';' is a literal
a ; echo b

$ cmd=echo
$ "$cmd" a ; $(printf echo) b  # ';' is a metacharacter
a
b

$ cmd='echo a'  # simple command is executed properly
$ $cmd
a

$ cmd='echo a;'  # but metacharacters are not treated
$ $cmd echo b
a; echo b

$ echo a $(echo ';') echo b  # ';' is a literal
a ; echo b

$ $(printf if) true; then echo a; fi  # error while parsing
bash: syntax error near unexpected token 'then'

$ $(printf if) true; $(printf then) echo a; $(printf fi)  # command not found
bash: if: command not found
bash: then: command not found
bash: fi: command not found

How to perform the command substitution for list of commands stored in a variable without using eval and bash -c (since eval is evil)?

Probalby, the right answer is "this is evil too" so we don't need to avoid eval :)

Variables hold data. Functions hold code. Don't put code inside variables!..

— from the article I'm trying to put a command in a variable, but the complex cases always fail!

Links

Unix stackexchange:

Bash shell command substitution

Why is a variable visible in a subshell?

Do parentheses really put the command in a subshell?

Other:

Bash Reference Manual

I'm trying to put a command in a variable, but the complex cases always fail!

conversation in debian bug tracking system

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.