3

From POSIX 7:

The order of word expansion shall be as follows:

  1. Tilde expansion (see Section 2.6.1), parameter expansion (see Section 2.6.2), command substitution (see Section 2.6.3), and arithmetic expansion (see Section 2.6.4) shall be performed, beginning to end. See item 5 in Section 2.3.

  2. Field splitting (see Section 2.6.5) shall be performed on the portions of the fields generated by step 1, unless IFS is null.

  3. Pathname expansion (see Section 2.6.6) shall be performed, unless set −f is in effect.

  4. Quote removal (see Section 2.6.7) shall always be performed last.

  1. Do tilde expansion, parameter expansion, command substitution, and arithmetic expansion perform in the specified order?

    Does the order between them matter? If yes, how shall we understand why the order is as specified?

  2. Why does pathname expansion happen after field splitting, while other expansions before field splitting?

    In particular, both tilde expansion and pathname expansion are about pathnames and filenames, why are they placed differently with respect to field splitting?

  3. Is there no brace expansion in POSIX?

  4. I notice "word expansion". Do expansions apply only to tokens with token identifier WORD, and not to tokens with other token identifiers (e.g. NAME, specific operator, NEWLINE, IO_NUMBER, ASSIGNMENT)?

3

Tilde expansion, parameter expansion, command substitution and arithmetic expansion are listed in the same step. That means that they are performed at the same time. The result of tilde expansion does not undergo parameter expansion, the result of parameter expansion does not undergo tilde expansion, and so on. For example, if the value of foo is $(bar) qux, then the word $foo expands to $(bar) qux at step 1; the text resulting from parameter expansion is not subject to any further transformation at step 1, but it then gets split by step 2.

“Beginning to end” means left-to-right processing, which matters e.g. when assignments occur: a=1; echo $a$((a=2))$a prints 122, because arithmetic expansion of $((a=2)) is performed, setting a to 2, between the parameter expansion of the first $a and the parameter expansion of the second $a.

The reason for the order is historical usage. POSIX usually follows existing implementation, it rarely specifies new behavior. There are multiple shells around; for the most part, POSIX follows the Korn shell but omits most features that are not present in the Bourne shell (as the Bourne shell is largely abandoned, the next version of POSIX is likely to include new ksh features though).

The reason why the Bourne shell performed parameter expansion then field splitting then globbing is that it allowed a glob to be stored in a variable: you can set a to *.txt *.pdf and then use $a to stand for the list of names of files matching *.txt followed by the list of names matching *.pdf (assuming both patterns match). (I'm not saying this is the best design possible, just that it was designed this way.) It's less clear to me why one would want command substitution to be placed at a particular step in the Bourne shell; in the Korn shell, its syntax $(…) is close to parameter expansion ${…} so it makes sense to perform them together.

The placement of tilde expansion is a historical oddity. It would have made more sense to place it later, so that you could write ~$some_user and have it expand to the home directory of the user whose name is the value of the variable some_user. I don't know why it wasn't done this way. This order even requires a special statement that the result of tilde expansion does not undergo other expansions (going by the passage you quoted, if HOME is /foo bar then ~ would expand to the two words /foo and bar due to field splitting, but no shell does that and POSIX.2008 explicitly states that “the pathname resulting from tilde expansion shall be treated as if quoted”).

There is no brace expansion in POSIX, otherwise the specification would state it.

Word expansion is only performed on WORDs, and with caveats mentioned in the following sections (e.g. field splitting and pathname generation are only performed in contexts that allow multiple words, not e.g. between double quotes). NAMEs, NEWLINEs, IO_NUMBERs and so on don't contain anything that could be expanded anyway.

  • Thanks. (1) In POSIX 2013, I found "The pathname resulting from tilde expansion shall be treated as if quoted to prevent it being altered by field splitting and pathname expansion." In your reply, "the result of tilde expansion does not undergo other expansions" is more stronger a statement than the one in POSIX, because "other expansions" are not only "field splitting and filename expansion". Which statement of the two is correct? – Tim Mar 17 '16 at 7:25
  • (2) I didn't find any similar statement to "The pathname resulting from tilde expansion shall be treated as if quoted to prevent it being altered by field splitting and pathname expansion" in Bash Reference Manual. Does Bash follows POSIX? – Tim Mar 17 '16 at 7:25
  • (3) Could you elaborate a little more about "you can set a=*.txt *.pdf and then use $a to stand for the list of names of files matching *.txt followed by the list of names matching *.pdf (assuming both patterns match)." I don't quite understand what you meant. In Bash, a=*.txt *.pdf outputs *.pdf: command not found, probably because there is a space betw a=*.txt and *.pdf. But I heard in variable assignment name=[value], value doesn't undergo word splitting and filename expansion. So I am not sure what a=*.txt *.pdf mean. – Tim Mar 17 '16 at 7:31
  • @Tim (1) Both are correct. (2) Yes. (3) There was an typo in my post, but please read for comprehension. – Gilles 'SO- stop being evil' Mar 17 '16 at 9:40
  • (4) Does "NAME" mean identifier? What does "WORD" mean, besides knowing it is exactly token to be expanded? (But we need to know what tokens are WORD, so that we know what tokes are to expand and what are not, don't we?) What does "ASSIGNMENT_WORD" mean? I read Shell Grammar in POSIX 2013 and some compiler book, but can't figure them out. See unix.stackexchange.com/questions/270275/… – Tim Mar 17 '16 at 23:03
2

Regarding (1), the order is based on existing implementation. If the people involved in writing it knew of different ordering for different shells that were deemed standard, they would provide some leeway in the wording to allow for this. As worded in 2.6 Word Expansions, there is no hint that other ordering could comply with the standard.

Again, with (2), keep in mind that this is based on existing implementation. The reasons for the original implementation are not necessarily found in the standard, although for some interesting areas of disagreement (e.g., BSD versus SVr4), a rationale section is provided to summarize the reason why the committee chose to pick one of the conflicting alternatives.

For (3), what you mention as "brace expansion" could be the same as 2.6.2 Parameter Expansion (though there are other possibilities, not necessarily in POSIX).

Finally (4), recall that a NAME is a certain type of WORD. Word-expansion would apply to names.

2
  1. First four expansions order:

The trigger for each expansion is distinct: Tilde expansion ~, parameter expansion $name and ${name}, command substitution $() and ` `, and arithmetic expansion $((...))). So the order does not matter much as there is no way to confuse one for the other, and each expansion is performed only once. A token that has been subject to parameter expansion will not be subject to command substitution (for example).
One expansion that may confuse things is the arithmetic expansion because (inside it) there may be parameter expansion, expansion, string expansion, command substitution, and quote removal. And, Arithmetic expansions may be nested. But all that happens inside an already detected arithmetic expansion, not outside.

The order that mostly matter is left to right.

After all that, the results of the above expansions (if not quoted) will be subject to field splitting and pathname expansion (in that order).

All expansions will be subject to quote removal at last.

  1. No, there is no brace expansion in POSIX.
  2. Word and token are usually used to mean the same idea in this case.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.