Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Or, an introductory guide to robust filename handling and other string passing in shell scripts.

I wrote a shell script which works well most of the time. But it chokes on some inputs (e.g. on some file names) — I encountered a problem such as the following:

  • I have a file name containing a space hello world, and it was treated as two separate files hello and world.
  • I have an input line with two consecutive spaces and they shrank to one in the input.
  • Leading and trailing whitespace disappears from input lines.
  • Sometimes, when the input contains one of the characters \[*?, they are replaced by some text which is is actually the name of files.
  • There is an apostrophe ' (or a double quote ") in the input and things got weird after that point.
  • There is a backslash in the input (or: I am using Cygwin and some of my file names have Windows-style \ separators).

What is going on and how do I fix it?

share|improve this question

2 Answers 2

up vote 25 down vote accepted

Always use double quotes around variable substitutions and command substitutions: "$foo", "$(foo)"

If you use $foo unquoted, your script will choke on input or parameters (or command output, with $(foo)) containing whitespace or \[*?.

There, you can stop reading. Well, ok, here are a few more:

  • readTo read input line by line with the read builtin, use while IFS= read -r line; do …
    Plain read treats backslashes and whitespace specially.
  • xargsAvoid xargs. If you must use xargs, make that xargs -0. Instead of find … | xargs, prefer find … -exec ….
    xargs treats whitespace and the characters \"' specially.

This answer applies to Bourne/POSIX-style shells (sh, ash, dash, bash, ksh, mksh, …). Zsh users should skip it and read the end of When is double-quoting necessary? instead. If you want the whole nitty-gritty, read the standard or your shell's manual.


Note that the explanations below contains a few approximations (statements that are true in most conditions but can be affected by the surrounding context or by configuration).

Why do I need to write "$foo"? What happens without the quotes?

$foo does not mean “take the value of the variable foo”. It means something much more complex:

  • First, take the value of the variable.
  • Field splitting: treat that value as a whitespace-separated list of fields, and build the resulting list. For example, if the variable contains foo * bar ​ then the result of this step is the 3-element list foo, *, bar.
  • Filename generation: treat each field as a glob, i.e. as a wildcard pattern, and replace it by the list of file names that match this pattern. If the pattern doesn't match any files, it is left unmodified. In our example, this results in the list containing foo, following by the list of files in the current directory, and finally bar. If the current directory is empty, the result is foo, *, bar.

Note that the result is a list of strings. There are two contexts in shell syntax: list context and string context. Field splitting and filename generation only happen in list context, but that's most of the time. Double quotes delimit a string context: the whole double-quoted string is a single string, not to be split. (Exception: "$@" to expand to the list of positional parameters, e.g. "$@ is equivalent to "$1" "$2" "$3" if there are three positional parameters. See What is the difference between $* and $@?)

The same happens to command substitution with $(foo) or with `foo`. On a side note, don't use `foo`: its quoting rules are weird and non-portable, and all modern shells support $(foo) which is absolutely equivalent except for having intuitive quoting rules.

The output of arithmetic substitution also undergoes the same expansions, but that isn't normally a concern as it only contains non-expandable characters (assuming IFS doesn't contain digits or -).

See When is double-quoting necessary? for more details about the cases when you can leave out the quotes.

Unless you mean for all this rigmarole to happen, just remember to always use double quotes around variable and command substitutions.

How do I process a list of file names?

If you write myfiles="file1 file2", with spaces to separate the files, this can't work with file names containing spaces. Unix file names can contain any character other than / (which is always a directory separator) and null bytes (which you can't use in shell scripts with most shells).

Same problem with myfiles=*.txt; … process $myfiles. When you do this, the variable myfiles contains the 5-character string *.txt, and it's when you write $myfiles that the wildcard is expanded. This example will actually work, until you change your script to be myfiles="$someprefix*.txt"; … process $myfiles. If someprefix is set to final report, this won't work.

To process a list of any kind (such as file names), put it in an array. This requires mksh, ksh93 or bash (or zsh, which doesn't have all these quoting issues); a plain POSIX shell (such as ash, dash or yash) doesn't have array variables.

myfiles=("$someprefix"*.txt)
process "${myfiles[@]}"

Ksh88 has array variables with a different assignment syntax set -A myfiles "someprefix"*.txt (see assignation variable under different ksh environment if you need ksh88/bash portability). Bourne/POSIX-style shells have a single one array, the array of positional parameters "$@" which you set with set and which is local to a function:

set -- "$someprefix"*.txt
process "$@"

What's up with read?

Without -r, read allows continuation lines — this is a single logical line of input:

hello \
world

read splits the input line into fields delimited by characters in $IFS (without -r, backslash also escapes those). For example, if the input is a line containing three words, then read first second third sets first to the first word of input, second to the second word and third to the third word. If there are more words, the last variable contains everything that's left after setting the preceding ones. Leading and trailing whitespace are trimmed.

Setting IFS to the empty string avoids any trimming. See Why is `while IFS= read` used so often, instead of `IFS=; while read..`? for a longer explanation.

What's wrong with xargs?

The input format of xargs is whitespace-separated strings which can optionally be single- or double-quoted. No standard tool outputs this format.

The input to xargs -L1 or xargs -l is almost a list of lines, but not quite — if there is a space at the end of a line, the following line is a continuation line.

You can use xargs -0 where applicable (and where available: GNU (Linux, Cygwin), BusyBox, BSD, OSX, but it isn't in POSIX). That's safe, because null bytes can't appear in most data, in particular in file names. To produce a null-separated list of file names, use find … -print0 (or you can use find … -exec … as explained below).

How do I process files found by find?

find … -exec some_command a_parameter another_parameter {} +

some_command needs to be an external command, it can't be a shell function or alias. If you need to invoke a shell to process the files, call sh explicitly.

find … -exec sh -c '
  for x do
    … # process the file "$x"
  done
' find-sh {} +

I have some other question

Browse the tag on this site, or or . (Click on “learn more…” to see some general tips and a hand-selected list of common questions.) If you've searched and you can't find an answer, ask away.

share|improve this answer
    
+1 Nicely done. I have just one addition regarding "The input format of xargs is whitespace-separated strings which can optionally be single- or double-quoted. No standard tool outputs this format": GNU's ls --quoting-style=shell-always provides this format and often seems useful. –  John1024 May 24 at 5:00
1  
@John1024 It's a GNU feature only, so I'll stick with “no standard tool”. –  Gilles May 24 at 5:02
    
You also need quotes around $(( ... )) (also $[...] in some shells) except in zsh (even in sh emulation) and mksh. –  Stéphane Chazelas May 24 at 6:39
    
Note that xargs -0 is not POSIX. Except with FreeBSD xargs, you generally want xargs -r0 instead of xargs -0. –  Stéphane Chazelas May 24 at 6:41
    
@John1024, no, ls --quoting-style=shell-always is not compatible with xargs. Try touch $'a\nb'; ls --quoting-style=shell-always | xargs –  Stéphane Chazelas May 24 at 6:50

While Gilles answer is excellent, I take issue at his main point

Always use double quotes around variable substitutions and command substitutions: "$foo", "$(foo)"

When you are starting out with a Bash-like shell that does word splitting, yes of course the safe advice is always use quotes. However word splitting is not always performed

§ Word Splitting

These commands can be run without error

foo=$bar
bar=$(a command)
logfile=$logdir/foo-$(date +%Y%m%d)
PATH=/usr/local/bin:$PATH ./myscript

I am not encouraging users to adopt this behavior, but if someone firmly understands when word splitting occurs then they should be able to decide for themselves when to use quotes.

share|improve this answer
1  
As I mention in my answer, see unix.stackexchange.com/questions/68694/… for details. Do notice the question — “Why does my shell script choke?”. The most common problem (from years of experience on this site and elsewhere) is missing double quotes. “Always use double quotes” is easier to remember than “always use double quotes, except for these cases where they aren't necessary”. –  Gilles May 24 at 8:25
1  
@Gilles - that's a good point, but I think it's good Steven said his thing too, though. Especially important I think - though not mentioned here - is the ability to parse shell argument arrays by quoting then unquoting $* after defining $IFS different ways. And, though less importantly probably, since you do touch on globbing, maybe mentioning set [-+]f could be worthwhile. –  mikeserv May 24 at 8:28
1  
@Gilles yes while Always ... except ... is not as pithy, it is more correct. –  Steven Penny May 24 at 16:17
1  
Rules are difficult to understand for beginners. For instance, foo=$bar is OK, but export foo=$bar or env foo=$var are not (at least in some shells). An advise for beginner: always quote your variables unless you know what you're doing and have a good reason not to. –  Stéphane Chazelas May 24 at 17:05
1  
@StephaneChazelas - it's because they're difficult to understand that rules need repeating - not sidelining. –  mikeserv May 25 at 2:22

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.