Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I don't understand shell expansion fully yet (hopefully, one day soon I will)...
I saw this comment to a superuser question, but I think I'm still parked at the kerb...

Using Linux without the shell is like driving a Ferrari at 50 km/h through city traffic. All fun will just go away ...

I don't understand the following example.. What heirarchy, or whatever, is causing the 2nd example "array item count:" to be different to the 1st example?

What happened to the shell introduced "space"?. or is it the echo which is introducing the space, and the shell is (perhaps) using \0?

#!/bin/bash
# Make a couple of files whose names contain a space.
junkd=$HOME/junkd
mkdir $junkd # || exit 1
cd $junkd
touch f\ {1..2}
#
echo -n * |xxd         # This shows a space between the two names.
names=$(echo -n * )
echo -n "$names" |xxd  # This shows a space between the two names.
#
# So far, it seems that the shell is inserting a space between each filename.
#
array=( $names )
echo "array item count: ${#array[@]}" 
# 4 items... This shows that a space is the delimiter char ....
#
array=( * )
echo "array item count: ${#array[@]}" 
# 2 items... What happened to the shell introduced space?
#
share|improve this question
add comment

2 Answers

up vote 2 down vote accepted

From man bash

EXPANSION Expansion is performed on the command line after it has been split into words. There are seven kinds of expansion performed: brace expansion, tilde expansion, parameter and variable expansion, command substitution, arithmetic expansion, word splitting, and pathname expansion. The order of expansions is: brace expansion, tilde expansion, parameter, variable and arithmetic expansion and command substitution (done in a left-to-right fashion), word splitting, and pathname expansion.

array=( $names )

The reason this gives you 4 entries is because an unquoted $names parameter is further subject to word splitting based on the internal field separator IFS which is by default <space><tab><newline>. If you were to quote "$names" to inhibit word splitting, then you'll only get one array element with value f 1 f 2, again not what you want.

array=( * )

The above on the other hand is only subject to pathname expansion which happens to be the last expansion performed. The results are not subject to word splitting thus you get the desired 2 elements.

If you want to make array=( $names ) work then you'll need to somehow separate the file names by a non-space character which also is not contained in the file names. You'll then need to set IFS to this character.

$ names=$(echo f* | sed "s/ /#/2")
$ echo $names
f 1#f 2
$ IFS='#' array=( $names )
$ echo ${#array[@]}
2
$ echo ${array[0]}
f 1

A more elegant way to do this would be to use the NUL byte \0 as the filename delimiter as that is guaranteed to never be apart of a filename. To accomplish this we will need to use the find command with its -print0 flag as well as the read builtin delimited on NUL. We well also need to clear IFS so no word splitting on spaces is performed.

#!/bin/bash

unset array

while IFS= read -r -d $'\0' name; do
  array+=( "$name" )
done < <(find . -type f -name "f*" -print0 )

Update

Expansion is performed on the command line after it has been split into words.

I can see how one would be confused by the quote above only to have it further state that word splitting is the 2nd to last expansion to occur.

A better way to word that quote in my opinion would be:

Expansion is performed on the command line after it has been split into arguments.

The splitting of arguments on the shell is always done by white space, and it's those arguments which are further subject to expansion. If you want to have white space in your argument you must either use Quoting or Escaping. IFS does not augment argument splitting, only word splitting.

Consider this example:

$ touch f{1,2}; IFS="#"; rm f1#f2
rm: cannot remove `f1#f2': No such file or directory

Notice how setting IFS to # did not alter the fact that the shell still only saw one argument f1#f2; which by the way is then further subject to the various expansions.

I would highly recommend your aquatint yourself with the BashFAQ if you haven't already. In particular, I would strongly suggest you read the following two supplemental entries:

  1. Arguments
  2. Word Splitting
share|improve this answer
    
I'm slowly getting there. I've started reading the entire man bash (10% so far :). Now, about this issue in particular. I'm starting to get a feel for it, but unless my compehension is way off, the man page excerpt you quoted, says, "Expansion is performed on the command line after it has been split into words..." and then goes on to say that one of the "expansions" is word splitting. That looks like a contradiction.. so it brings up the "mechanism" issue again.. I've just noticed Gilles has posted and mentions something about "shell words".. (but I must go now... back later) –  Peter.O Mar 12 '11 at 0:43
    
@fred.bear please see my updated answer, this should put to rest any doubt you may have had regarding the "mechanics" of the shell. –  SiegeX Mar 12 '11 at 1:35
    
I'm sure the answer is probably glaringly obvious in both yours and Gilles' answer, but untill I can actually see it, I can't assess the 'best' answer.. They are both good answers and have got me on the right track ... I'm about to go through your updated answer now... Perhaps I'm missing the point because I've been thinking that the only thing that "word splitting" did was to remove duplicate spaces between any and all "words", but I'm starting to think that filename expansion creates a very non-human word (ie. a word which includes spaces), otherwise how does the array assignment work? –  Peter.O Mar 12 '11 at 4:56
    
Introducing the word "argument", won the argument. It tallies with my evolving line of thought, so I probably do understand it now...(I just need to put it into practice)... It did however take two of you to get through my thick scull on this one, and I thank you both for the information in both answers.. I've learnt different things from both. –  Peter.O Mar 12 '11 at 5:18
add comment

A shell command (more precisely, a “simple command”) consists of a list of words. Each word can be an arbitrary string (a shell word can contain spaces and punctuation characters).

When you run echo -n *, the shell performs pathname expansion (also called filename generation or globbing) on *, and replaces it by the list of matching file names. So after expansion, this command consists of four words: echo, -n, f 1 and f 2. The command echo is run with two arguments, and it prints its arguments with a space in between (and no terminating newline because of the -n option). So the output is f 1 f 2. Exercise: create another file whose name consists of two consecutive spaces, run echo -n *, and make sure you understand the output.

When you run names=$(echo -n * ), the output from the command is stored in the names variable. Here, that line is equivalent to names='f 1 f 2'.

Now we get to array=( $names ). That's an array assignment, but it doesn't affect the expansion in this case. Since $names is an unquoted variable expansion, it's subject to word splitting followed by pathname expansion. Word splitting means that the value of the variable (which is a string) is split into pieces at each whitespace sequence (for the precise rules, search IFS in your shell's documentation). You can end up with zero, one or more words; here the string is split into 4 words: f, 1, f and 2. So the array contains four elements (each a one-character word). Exercise: with that extra file with two consecutive spaces in its name, what is now the exact contents of the array?

Next, you tried array=( * ). Here, there's a single word in the array, subject to the usual expansions, the last of which is pathname expansion. Since there are two matching files, the array contains two words, the names of each file: f 1 and f 2.

In terms of shell programming practice, what advice can we draw from this analysis? Well, first, there's the usual shell programming principle: always put double quotes around variable expansions, unless you have a good reason not to. Then, don't store a list in a string variable. If you want to store a list of file names, put it directly in an array:

files=(*)
ls -l "${files[@]}"

Further exercise: create a file whose name is a single asterisk (touch '*') and run these commands again. Do you understand the output?

Aside: zsh does not perform word splitting or pathname expansion on variable expansions. This makes it quite a bit saner to program in.

share|improve this answer
    
Was there something in my answer that you did not find sufficient or was this a point grab? Just curious –  SiegeX Mar 12 '11 at 1:00
    
@SiegeX: Apart from the useless diversion with # as a separator (which won't work if a file name contains #), your answer is correct. But it's not very didactic. In particular, you don't explain that a command is not a string but a list of strings, which is often confusing to shell beginners. –  Gilles Mar 12 '11 at 1:09
    
We'll have to agree to disagree on the # example because I explicitly used it to be didactic; It's much easier to visualize # than the NUL byte. I also made special note that this char must not be apart of the filename and then proceeded to give the more proper NUL byte solution. From my perspective, your answer was 90% the same and the other 10% would have been more appropriate as a comment. But, you can't get to 21K for making only comments so C'est la vie. –  SiegeX Mar 12 '11 at 1:26
    
@SiegeX: If you think my answer is 90% the same as yours, you need to work on your pedagogy. We are explaining the same facts very differently. –  Gilles Mar 12 '11 at 1:30
    
The pedantry of your diction is indubitability noted. –  SiegeX Mar 12 '11 at 1:47
show 6 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.