Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm getting confused about the role word splitting plays in zsh. I have not been exposed to this concept when programming in C, Python or MATLAB, and this has triggered my interest of why word splitting seems to be something specific to shell programming.

I have read about word splitting on this and other sites before, but haven't found a clear explanation of the concept. Wikipedia has a definition of word splitting but does not seem to have references on how it applies to Unix shells.

Here's an example of my confusion in zsh:

In the Z Shell FAQ, I read the following:

3.1: Why does $var where var="foo bar" not do what I expect?

In most Bourne-shell derivatives, multiple-word variables such as var="foo bar" are split into words when passed to a command or used in a for foo in $var loop. By default, zsh does not have that behaviour: the variable remains intact. (This is not a bug! See below.) The option SH_WORD_SPLIT exists to provide compatibility.

However, in the Z Shell Manual, I read the following:

SH_WORD_SPLIT (-y) <K> <S>

Causes field splitting to be performed on unquoted parameter expansions. Note that this option has nothing to do with word splitting. (See Parameter Expansion.)

Why does it say that SH_WORD_SPLIT has nothing to do with word splitting? Isn't word splitting precisely what this is all about?

share|improve this question
add comment

2 Answers

up vote 10 down vote accepted

Early shells had only a single data type: strings. But it is common to manipulate lists of strings, typically when passing multiple file names as arguments to a program. Another common use case for splitting is when a command outputs a list of results: the command's output is a string, but the desired data is a list of strings. To store a list of file names in a variable, you would put spaces between them. Then a shell script like this

files="foo bar qux"
myprogram $files

called myprogram with three arguments, as the shell split the string $files into words. At the time, spaces in file names were either forbidden or widely considered Not Done.

The Korn shell introduced arrays: you could store a list of strings in a variable. The Korn shell remained compatible with the then-established Bourne shell, so bare variable expansions kept undergoing word splitting, and using arrays required some syntactic overhead. You would write the snippet above

files=(foo bar qux)
myprogram "${files[@]}"

Zsh had arrays from the start, and its author opted for a saner language design at the expense of backward compatibility. In zsh (under the default expansion rules) $var does not perfom word splitting; if you want to store a list of words in a variable, you are meant to use an array; and if you really want word splitting, you can write $=var.

files=(foo bar qux)
myprogram $files

These days, spaces in file names are something you need to cope with, both because many users expect them to work and because many scripts are executed in security-sensitive contexts where an attacker may be in control of file names. So automatic word splitting is often a nuisance; hence my general advice to always use double quotes, i.e. write "$foo", unless you understand why you need word splitting in a particular use case. (Note that bare variable expansions undergo globbing as well.)

share|improve this answer
    
Thanks Gilles, this is really helpful! Is it correct to say that roughly speaking word splitting converts strings of the form "word1 word2 word3" into lists/arrays of the form "word1" "word2" "word3"? I have also updated the OP with a specific source of confusion in zsh. –  user815423426 Dec 13 '11 at 1:56
1  
@intrpc "Word splitting" is not splitting on natural language words but on $IFS characters. Hence "field splitting" is a better name. But "word splitting" is often used for this concept in shell literature. The zsh documentation is quibbling on words. –  Gilles Dec 14 '11 at 10:11
    
See also rc (the plan9 shell, also ported to Unix) for an even better design than zsh when it comes to variables and arrays. –  Stephane Chazelas Feb 8 '13 at 23:17
add comment

Word splitting is not really shell specific.

Most programs that need to parse text input use some form of word splitting as a first step. It is done before identifying from these "words", numbers, operators, strings, tokens and whatever similar entities they need to process.

What is specific with the shells is that they have to properly build the argument list of commands called (C argc/argv, python sys.argv), including passing arguments with embedded spaces, empty arguments, custom delimiters and so on. Many shells use the IFS variable to allow some flexibility there.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.