Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

The following few threads on this site and StackOverflow were helpful for understanding how IFS works:

But I still have some short questions. I decided to ask them in the same post since I think it may help better future readers:

Q1. IFS is typically discussed in the context of "field splitting". Is field splitting the same as word splitting ?

Q2: The POSIX specification says:

If the value of IFS is null, no field splitting shall be performed.

Is setting IFS= the same as setting IFS to null? Is this what is meant by setting it to an empty string too?

Q3: In the POSIX specification, I read the following:

If IFS is not set, the shell shall behave as if the value of IFS is <space>, <tab> and <newline>

Say I want to restore the default value of IFS. How do I do that? (more specifically, how do I refer to <tab> and <newline>?)

Q4: Finally, how would this code:

while IFS= read -r line
do    
    echo $line
done < /path_to_text_file

behave if we we change the first line to

while read -r line # Use the default IFS value

or to:

while IFS=' ' read -r line
share|improve this question

4 Answers 4

up vote 10 down vote accepted
  1. Yes, they are the same.
  2. Yes.
  3. In bash, and similar shells, you could do something like IFS=$' \t\n'. Otherwise, you could insert the literal control codes by using [space] CTRL+V [tab] CTRL+V [enter]. If you are planning to do this, however, it's better to use another variable to temporarily store the old IFS value, and then restore it afterwards (or temporarily override it for one command by using the var=foo command syntax).
    • The first code snippet will put the entire line read, verbatim, into $line, as there are no field separators to perform word splitting for. Bear in mind however that since many shells use cstrings to store strings, the first instance of a NUL may still cause the appearance of it being prematurely terminated.
    • The second code snippet may not put an exact copy of the input into $line. For example, if there are multiple consecutive field separators, they will be made into a single instance of the first element. This is often recognised as loss of surrounding whitespace.
    • The third code snippet will do the same as the second, except it will only split on a space (not the usual space, tab, or newline).
share|improve this answer
2  
The answer to Q2 is wrong: an empty IFS and an unset IFS are very different. The answer to Q4 is partly wrong: inner separators are not touched here, only leading and trailing ones. –  Gilles Dec 14 '11 at 12:06
2  
@Gilles: In Q2 none of the three given denominations refers to an unset IFS, all of them mean IFS=. –  Stéphane Gimenez Dec 14 '11 at 13:00
    
@Gilles In Q2, I never said they were the same. And inner separators are touched, as shown here: IFS=' ' ; foo=( bar baz qux ) ; echo "${#foo[@]}". (Er, what? There should be multiple space delimiters in there, SO engine keeps on stripping them). –  Chris Down Dec 14 '11 at 16:05
2  
@StéphaneGimenez, Chris: Oh, right, sorry about Q2, I misread the question. For Q4, we're talking about read; the last variable grabs everything that's left except for the last separator and leaves inner separators inside. –  Gilles Dec 15 '11 at 15:25
    
Gilles is partially correct about the spaces not being removed by read. Read my answer for details. –  BinaryZebra Aug 6 at 23:28

Q1: Yes. “Field splitting” and “word splitting” are two terms for the same concept.

Q2: Yes. If IFS is unset (i.e. after unset IFS), it is equivalent IFS being set to $' \t\n' (a space, a tab and a newline). If IFS is set to an empty value (that's what “null” means here) (i.e. after IFS= or IFS='' or IFS=""), no field splitting is performed at all (and $*, which normally uses the first character of $IFS, uses a space character).

Q3: If you want to have the default IFS behavior, you can use unset IFS. If you want to set IFS explicitly to this default value, you can put the literal characters space, tab, newline in single quotes. In ksh93, bash or zsh, you can use IFS=$' \t\n'. Portably, if you want to avoid having a literal tab character in your source file, you can use

IFS=" $(echo t | tr t \\t)
"

Q4: With IFS set to an empty value, read -r line sets line to the whole line except its terminating newline. With IFS=" ", spaces at the beginning and at the end of the line are trimmed. With the default value of IFS, tabs and spaces are trimmed.

share|improve this answer
2  
Q2 is partly wrong. If IFS is empty, "$*" is joined without separators. (for $@, there are some variations between shells in non-list contexts like IFS=; var=$@). It should be noted that when IFS is empty, no word splitting is perfomed but $var still expands to no argument instead of an empty argument when $var is empty, and globbing still applies, so you still need to quote variables (even if you disable globbing) –  Stéphane Chazelas Feb 8 '13 at 22:19
    
I do not get the same in Q4 with IFS=" ", the spaces at the end of the line are not trimmed. Please read my answer. –  BinaryZebra Aug 7 at 4:31

unset IFS does clear IFS, even if IFS is thereafter presumed to be " \t\n":

$ echo "'$IFS'"
'   
'
$ IFS=""
$ echo "'$IFS'"
''
$ unset IFS
$ echo "'$IFS'"
''
$ IFS=$' \t\n'
$ echo "'$IFS'"
'   
'
$

Tested on bash versions 4.2.45 and 3.2.25 with the same behavior.

share|improve this answer

Q1. Field splitting.

Is field splitting the same as word splitting ?

Yes, both point to the same idea.

Q2: When is IFS null?.

Is setting IFS='' the same as null, the same as an empty string too?

Yes, all three mean the same: No field/word splitting shall be performed. Also, this affects printing fields (as with echo "$*") all fields will be concatenated together with no space.

Q3: (part a) Unset IFS.

In the POSIX specification, I read the following:

If IFS is not set, the shell shall behave as if the value of IFS is <space><tab><newline>.

Which is exactly equivalent to:

With an unset IFS, the shell shall behave as if IFS is default.

That means that the 'Field splitting' will be exactly the same with a default IFS value, or unset.
That does NOT mean that IFS will work the same way in all conditions. Being more specific, executing OldIFS=$IFS will set the var OldIFS to null, not the default. And trying to set IFS back, as this, IFS=OldIFS will set IFS to null, not keep it unset as it were before. Watch out !!.

Q3: (part b) Restore IFS.

How could I restore the value of IFS to default. Say I want to restore the default value of IFS. How do I do that? (more specifically, how do I refer to <tab> and <newline>?)

For zsh, ksh, and bash (AFAIK), IFS could be set to the default value as:

IFS=$' \t\n'        # works with zsh, ksh, bash.

Done, you need to read nothing else.

But if you need to re-set IFS for sh, it may become complex.

Let's take a look from easiest to complete with no drawbacks (except complexity).

1.- Unset IFS.

We could just unset IFS (Read Q3 part a, above.).

2.- Swap chars.

As a workaround, swapping the value of tab and newline makes it simpler to set the value of IFS, and then it works in a equivalent way.

Set IFS to <space><newline><tab>:

sh -c 'IFS=$(echo " \n\t"); printf "%s" "$IFS"|xxd'      # Works.

3.- A simple? solution:

If there are child scripts that need IFS correctly set, you could always manually write:

IFS='   
'

Where the sequence manually typed was: IFS='spacetabnewline', sequence which has actually been correctly typed above (If you need to confirm, edit this answer). But a copy/paste from your browser will break because the browser will squeeze/hide the whitespace. It makes it difficult to share the code as written above.

4.- Complete solution.

To write code that can be safely copied usually involves unambiguous printable escapes.

We need some code that "produces" the expected value. But, even if conceptually correct, this code will NOT set a trailing \n:

sh -c 'IFS=$(echo " \t\n"); printf "%s" "$IFS"|xxd'      # wrong.

That happens because, under most shells, all trailing newlines of $(...) or `...` command substitutions are removed on expansion.

We need to use a trick for sh:

sh -c 'IFS="$(printf " \t\nx")"; IFS="${IFS%x}"; printf "$IFS"|xxd'  # Correct.

An alternative way may be to set IFS as an environment value from bash (for example) and then call sh (the versions of it that accept IFS to be set via the environment), as this:

env IFS=$' \t\n' sh -c 'printf "%s" "$IFS"|xxd'

In short, sh makes resetting IFS to default quite an odd adventure.

Q4: In actual code:

Finally, how would this code:

while IFS= read -r line
do
    echo $line
done < /path_to_text_file

behave if we we change the first line to

while read -r line # Use the default IFS value

or to:

while IFS=' ' read -r line

First: I do not know if the echo $line (with the var NOT quoted) is there on porpouse, or not. It introduces a second level of 'field splitting' that read does not have. So I'll answer both. :)

With this code (so you could confirm). You'll need the useful xxd:

#!/bin/ksh
# Correctly set IFS as described above.
defIFS="$(printf " \t\nx")"; defIFS="${defIFS%x}";
IFS="$defIFS"
printf "IFS value: "
printf "%s" "$IFS"| xxd -p

a='   bar   baz   quz   '; l="${#a}"
printf "var value          : %${l}s-" "$a" ; printf "%s\n" "$a" | xxd -p

printf "%s\n" "$a" | while IFS='x' read -r line; do
    printf "IFS --x--          : %${l}s-" "$line" ;
    printf "%s" "$line" |xxd -p; done;

printf 'Values      quoted :\n' ""  # With values quoted:
printf "%s\n" "$a" | while IFS='' read -r line; do
    printf "IFS null    quoted : %${l}s-" "$line" ;
    printf "%s" "$line" |xxd -p; done;

printf "%s\n" "$a" | while IFS="$defIFS" read -r line; do
    printf "IFS default quoted : %${l}s-" "$line" ;
    printf "%s" "$line" |xxd -p; done;

unset IFS; printf "%s\n" "$a" | while read -r line; do
    printf "IFS unset   quoted : %${l}s-" "$line" ;
    printf "%s" "$line" |xxd -p; done;
    IFS="$defIFS"   # set IFS back to default.

printf "%s\n" "$a" | while IFS=' ' read -r line; do
    printf "IFS space   quoted : %${l}s-" "$line" ;
    printf "%s" "$line" |xxd -p; done;

printf '%s\n' "Values unquoted :"   # Now with values unquoted:
printf "%s\n" "$a" | while IFS='x' read -r line; do
    printf "IFS --x-- unquoted : "
    printf "%s, " $line; printf "%s," $line |xxd -p; done

printf "%s\n" "$a" | while IFS='' read -r line; do
    printf "IFS null  unquoted : ";
    printf "%s, " $line; printf "%s," $line |xxd -p; done

printf "%s\n" "$a" | while IFS="$defIFS" read -r line; do
    printf "IFS defau unquoted : ";
    printf "%s, " $line; printf "%s," $line |xxd -p; done

unset IFS; printf "%s\n" "$a" | while read -r line; do
    printf "IFS unset unquoted : ";
    printf "%s, " $line; printf "%s," $line |xxd -p; done
    IFS="$defIFS"   # set IFS back to default.

printf "%s\n" "$a" | while IFS=' ' read -r line; do
    printf "IFS space unquoted : ";
    printf "%s, " $line; printf "%s," $line |xxd -p; done

I get:

$ ./stackexchange-Understanding-IFS.sh
IFS value: 20090a
var value          :    bar   baz   quz   -20202062617220202062617a20202071757a2020200a
IFS --x--          :    bar   baz   quz   -20202062617220202062617a20202071757a202020
Values      quoted :
IFS null    quoted :    bar   baz   quz   -20202062617220202062617a20202071757a202020
IFS default quoted :       bar   baz   quz-62617220202062617a20202071757a
IFS unset   quoted :       bar   baz   quz-62617220202062617a20202071757a
IFS space   quoted :       bar   baz   quz-62617220202062617a20202071757a
Values unquoted :
IFS --x-- unquoted : bar, baz, quz, 6261722c62617a2c71757a2c
IFS null  unquoted : bar, baz, quz, 6261722c62617a2c71757a2c
IFS defau unquoted : bar, baz, quz, 6261722c62617a2c71757a2c
IFS unset unquoted : bar, baz, quz, 6261722c62617a2c71757a2c
IFS space unquoted : bar, baz, quz, 6261722c62617a2c71757a2c

The first value is just the correct value of IFS='spacetabnewline'

Next line is all the hex values that the var $a has, and a newline '0a' at the end as it is going to be given to each read command.

The next line, for which IFS is null, does not perform any 'field spliting', but the newline is removed (as expected).

The next three lines, as IFS contains an space, remove the initial spaces and set the var line to the balance remaining.

The last four lines shows what an unquoted variable will do. The values will be split on the (several) spaces and will be printed as: bar,baz,qux,

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.