How does the shell filename expansion delimit items within a ( * ) list?

Question

I don't understand shell expansion fully yet (hopefully, one day soon I will)...
I saw this comment to a superuser question, but I think I'm still parked at the kerb...

Using Linux without the shell is like driving a Ferrari at 50 km/h through city traffic. All fun will just go away ...

I don't understand the following example.. What heirarchy, or whatever, is causing the 2nd example "array item count:" to be different to the 1st example?

What happened to the shell introduced "space"?. or is it the echo which is introducing the space, and the shell is (perhaps) using \0?

#!/bin/bash
# Make a couple of files whose names contain a space.
junkd=$HOME/junkd
mkdir $junkd # || exit 1
cd $junkd
touch f\ {1..2}
#
echo -n * |xxd         # This shows a space between the two names.
names=$(echo -n * )
echo -n "$names" |xxd  # This shows a space between the two names.
#
# So far, it seems that the shell is inserting a space between each filename.
#
array=( $names )
echo "array item count: ${#array[@]}" 
# 4 items... This shows that a space is the delimiter char ....
#
array=( * )
echo "array item count: ${#array[@]}" 
# 2 items... What happened to the shell introduced space?
#

Answer 1

From man bash

EXPANSION Expansion is performed on the command line after it has been split into words. There are seven kinds of expansion performed: brace expansion, tilde expansion, parameter and variable expansion, command substitution, arithmetic expansion, word splitting, and pathname expansion. The order of expansions is: brace expansion, tilde expansion, parameter, variable and arithmetic expansion and command substitution (done in a left-to-right fashion), word splitting, and pathname expansion.

array=( $names )

The reason this gives you 4 entries is because an unquoted $names parameter is further subject to word splitting based on the internal field separator IFS which is by default <space><tab><newline>. If you were to quote "$names" to inhibit word splitting, then you'll only get one array element with value f 1 f 2, again not what you want.

array=( * )

The above on the other hand is only subject to pathname expansion which happens to be the last expansion performed. The results are not subject to word splitting thus you get the desired 2 elements.

If you want to make array=( $names ) work then you'll need to somehow separate the file names by a non-space character which also is not contained in the file names. You'll then need to set IFS to this character.

$ names=$(echo f* | sed "s/ /#/2")
$ echo $names
f 1#f 2
$ IFS='#' array=( $names )
$ echo ${#array[@]}
2
$ echo ${array[0]}
f 1

A more elegant way to do this would be to use the NUL byte \0 as the filename delimiter as that is guaranteed to never be apart of a filename. To accomplish this we will need to use the find command with its -print0 flag as well as the read builtin delimited on NUL. We well also need to clear IFS so no word splitting on spaces is performed.

#!/bin/bash

unset array

while IFS= read -r -d $'\0' name; do
  array+=( "$name" )
done < <(find . -type f -name "f*" -print0 )

Update

Expansion is performed on the command line after it has been split into words.

I can see how one would be confused by the quote above only to have it further state that word splitting is the 2nd to last expansion to occur.

A better way to word that quote in my opinion would be:

Expansion is performed on the command line after it has been split into arguments.

The splitting of arguments on the shell is always done by white space, and it's those arguments which are further subject to expansion. If you want to have white space in your argument you must either use Quoting or Escaping. IFS does not augment argument splitting, only word splitting.

Consider this example:

$ touch f{1,2}; IFS="#"; rm f1#f2
rm: cannot remove `f1#f2': No such file or directory

Notice how setting IFS to # did not alter the fact that the shell still only saw one argument f1#f2; which by the way is then further subject to the various expansions.

I would highly recommend your aquatint yourself with the BashFAQ if you haven't already. In particular, I would strongly suggest you read the following two supplemental entries:

1. Arguments
2. Word Splitting

Answer 2

A shell command (more precisely, a “simple command”) consists of a list of words. Each word can be an arbitrary string (a shell word can contain spaces and punctuation characters).

When you run echo -n *, the shell performs pathname expansion (also called filename generation or globbing) on *, and replaces it by the list of matching file names. So after expansion, this command consists of four words: echo, -n, f 1 and f 2. The command echo is run with two arguments, and it prints its arguments with a space in between (and no terminating newline because of the -n option). So the output is f 1 f 2. Exercise: create another file whose name consists of two consecutive spaces, run echo -n *, and make sure you understand the output.

When you run names=$(echo -n * ), the output from the command is stored in the names variable. Here, that line is equivalent to names='f 1 f 2'.

Now we get to array=( $names ). That's an array assignment, but it doesn't affect the expansion in this case. Since $names is an unquoted variable expansion, it's subject to word splitting followed by pathname expansion. Word splitting means that the value of the variable (which is a string) is split into pieces at each whitespace sequence (for the precise rules, search IFS in your shell's documentation). You can end up with zero, one or more words; here the string is split into 4 words: f, 1, f and 2. So the array contains four elements (each a one-character word). Exercise: with that extra file with two consecutive spaces in its name, what is now the exact contents of the array?

Next, you tried array=( * ). Here, there's a single word in the array, subject to the usual expansions, the last of which is pathname expansion. Since there are two matching files, the array contains two words, the names of each file: f 1 and f 2.

In terms of shell programming practice, what advice can we draw from this analysis? Well, first, there's the usual shell programming principle: always put double quotes around variable expansions, unless you have a good reason not to. Then, don't store a list in a string variable. If you want to store a list of file names, put it directly in an array:

files=(*)
ls -l "${files[@]}"

Further exercise: create a file whose name is a single asterisk (touch '*') and run these commands again. Do you understand the output?

Aside: zsh does not perform word splitting or pathname expansion on variable expansions. This makes it quite a bit saner to program in.