26

Why do I get different values for $x from the snippets below?

#!/bin/bash

x=1
echo fred > junk ; while read var ; do x=55 ; done < junk
echo x=$x 
#    x=55 .. I'd expect this result

x=1
cat junk | while read var ; do x=55 ; done
echo x=$x 
#    x=1 .. but why?

x=1
echo fred | while read var ; do x=55 ; done
echo x=$x 
#    x=1  .. but why?
27

The right explanation has already been given by jsbillings and geekosaur, but let me expand on that a bit.

In most shells, including bash, each side of a pipeline runs in a subshell, so any change in the shell's internal state (such as setting variables) remains confined to that segment of a pipeline. The only information you can get from a subshell is what it outputs (to standard output and other file descriptors) and its exit code (which is a number between 0 and 255). For example, the following snippet prints 0:

a=0; a=1 | a=2; echo $a

In ksh (the variants derived from the AT&T code, not pdksh/mksh variants) and zsh, the last item in a pipeline is executed in the parent shell. (POSIX allows both behaviors.) So the snippet above prints 2.

A useful idiom is to include the continuation of the while loop (or whatever you have on the right-hand side of the pipeline, but a while loop is actually common here) in the pipeline:

cat junk | {
  while read var ; do x=55 ; done
  echo x=$x 
}
| improve this answer | |
  • 3
    Thanks Gilles .. That a=0; a=1 | a=2 gives a very clear picture.. and not only of the localization of internal state, but also that a pipeline doesn't actually need to send anything through the pipe (other than the exit code(?).. In itself that is an interesting insight into a pipe ... I did manage to get my script running with < <(locate -ber ^\.tag$), thanks to the original slightly unclear answer and geekosaur and glenn jackman's comemnts.. I was initially in a dilemma about accepting the answer, but the nett result was pretty clear, especially with jsbillings follow-up comment :) – Peter.O Mar 24 '11 at 12:31
  • it feels like I piped into a function, so I moved some variables and tests to inside it and it worked great, thx! – Aquarius Power Aug 1 '14 at 21:04
  • Syntax ... | while read item pipes the stdout of whatever is run before while but with this syntax the body of while is always run in a subshell. I prefer to use syntax for item in $(...) instead, because that keeps the body of the loop in current shell. – Mikko Rantalainen Apr 26 at 11:43
  • Unfortunately, the syntax for item in $(...) syntax does not nicely support all features. For example ... | while read col1 col2 col3 rest cannot be nicely implemented with for .. in syntax. – Mikko Rantalainen Apr 26 at 11:44
  • 1
    @MikkoRantalainen Beware that for item in $(…) doesn't do the same thing. It expands wildcards unless you disable that with set -f. It splits at all whitespace, not just newlines, unless you change IFS. It strips trailing blank lines. Don't use for item in $(…) unless you've thought of all of these things. – Gilles 'SO- stop being evil' Apr 26 at 11:46
9

You're running into a variable scope issue. The variables defined in the while loop that is on the right side of the pipe have their own local scope context, and changes to the variable will not be seen outside of the loop. The while loop is essentially a subshell which gets a COPY of the shell environment, and any changes to the environment are lost at the end of the shell. See this StackOverflow question.

UPDATED: I neglected to point out the important fact that the while loop with it's own subshell was due to it being the endpoint of a pipe, I've updated that in the answer.

| improve this answer | |
  • @jsbillings.. Okay, that explains the two last snippets, but it doesn't explain the first, where the value of $x set in the loop, is carried on as 55 (beyond the scope of the 'while' loop) – Peter.O Mar 23 '11 at 14:39
  • 5
    @fred.bear: It's running the while loop as the tail end of a pipeline that throws it into a subshell. – geekosaur Mar 23 '11 at 14:44
  • 2
    This is where bash process substitution comes into play. Instead of blah|blah|while read ..., you can have while read ...; done < <(blah|blah) – glenn jackman Mar 23 '11 at 15:01
  • 1
    @geekosaur: thanks for filling in the details that I neglected to include in my answer. – jsbillings Mar 23 '11 at 15:31
  • 1
    -1 Sorry but this answer is just wrong. It explains how this stuff works in many programming languages but not in the shell. @Gilles, below, got it right. – jpc Mar 24 '11 at 3:13
7

As mentioned in other answers, the parts of a pipeline run in subshells, so modifications made there aren't visible to the main shell.

If we consider just Bash, there are two other workarounds in addition to the cmd | { stuff; more stuff; } structure:

  1. Redirect the input from process substitution:

    while read var ; do x=55 ; done < <(echo fred)
    echo "$x"
    

    The output from the command in <(...) is made to appear as if it were a named pipe.

  2. The lastpipe option, which makes Bash work like ksh, and runs the last part of the pipeline in the main shell process. Though it only works if job control is disabled, i.e. not in an interactive shell:

    bash -c '
      shopt -s lastpipe
      echo fred | while read var ; do x=55 ; done; 
      echo "$x"
    '
    

    or

    bash -O lastpipe -c '
      echo fred | while read var ; do x=55 ; done; 
      echo "$x"
    '
    

Process substitution is of course supported in ksh and zsh too. But since they run the last part of the pipeline in the main shell anyway, using it as a workaround isn't really necessary.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.