22

Why do I get different values for $x from the snippets below?

#!/bin/bash

x=1
echo fred > junk ; while read var ; do x=55 ; done < junk
echo x=$x 
#    x=55 .. I'd expect this result

x=1
cat junk | while read var ; do x=55 ; done
echo x=$x 
#    x=1 .. but why?

x=1
echo fred | while read var ; do x=55 ; done
echo x=$x 
#    x=1  .. but why?
25

The right explanation has already been given by jsbillings and geekosaur, but let me expand on that a bit.

In most shells, including bash, each side of a pipeline runs in a subshell, so any change in the shell's internal state (such as setting variables) remains confined to that segment of a pipeline. The only information you can get from a subshell is what it outputs (to standard output and other file descriptors) and its exit code (which is a number between 0 and 255). For example, the following snippet prints 0:

a=0; a=1 | a=2; echo $a

In ksh (the variants derived from the AT&T code, not pdksh/mksh variants) and zsh, the last item in a pipeline is executed in the parent shell. (POSIX allows both behaviors.) So the snippet above prints 2.

A useful idiom is to include the continuation of the while loop (or whatever you have on the right-hand side of the pipeline, but a while loop is actually common here) in the pipeline:

cat junk | {
  while read var ; do x=55 ; done
  echo x=$x 
}
  • 1
    Thanks Gilles .. That a=0; a=1 | a=2 gives a very clear picture.. and not only of the localization of internal state, but also that a pipeline doesn't actually need to send anything through the pipe (other than the exit code(?).. In itself that is an interesting insight into a pipe ... I did manage to get my script running with < <(locate -ber ^\.tag$), thanks to the original slightly unclear answer and geekosaur and glenn jackman's comemnts.. I was initially in a dilemma about accepting the answer, but the nett result was pretty clear, especially with jsbillings follow-up comment :) – Peter.O Mar 24 '11 at 12:31
  • it feels like I piped into a function, so I moved some variables and tests to inside it and it worked great, thx! – Aquarius Power Aug 1 '14 at 21:04
9

You're running into a variable scope issue. The variables defined in the while loop that is on the right side of the pipe have their own local scope context, and changes to the variable will not be seen outside of the loop. The while loop is essentially a subshell which gets a COPY of the shell environment, and any changes to the environment are lost at the end of the shell. See this StackOverflow question.

UPDATED: I neglected to point out the important fact that the while loop with it's own subshell was due to it being the endpoint of a pipe, I've updated that in the answer.

  • @jsbillings.. Okay, that explains the two last snippets, but it doesn't explain the first, where the value of $x set in the loop, is carried on as 55 (beyond the scope of the 'while' loop) – Peter.O Mar 23 '11 at 14:39
  • 5
    @fred.bear: It's running the while loop as the tail end of a pipeline that throws it into a subshell. – geekosaur Mar 23 '11 at 14:44
  • 2
    This is where bash process substitution comes into play. Instead of blah|blah|while read ..., you can have while read ...; done < <(blah|blah) – glenn jackman Mar 23 '11 at 15:01
  • 1
    @geekosaur: thanks for filling in the details that I neglected to include in my answer. – jsbillings Mar 23 '11 at 15:31
  • 1
    -1 Sorry but this answer is just wrong. It explains how this stuff works in many programming languages but not in the shell. @Gilles, below, got it right. – jpc Mar 24 '11 at 3:13
5

As mentioned in other answers, the parts of a pipeline run in subshells, so modifications made there aren't visible to the main shell.

If we consider just Bash, there are two other workarounds in addition to the cmd | { stuff; more stuff; } structure:

  1. Redirect the input from process substitution:

    while read var ; do x=55 ; done < <(echo fred)
    echo "$x"
    

    The output from the command in <(...) is made to appear as if it were a named pipe.

  2. The lastpipe option, which makes Bash work like ksh, and runs the last part of the pipeline in the main shell process. Though it only works if job control is disabled, i.e. not in an interactive shell:

    bash -c '
      shopt -s lastpipe
      echo fred | while read var ; do x=55 ; done; 
      echo "$x"
    '
    

    or

    bash -O lastpipe -c '
      echo fred | while read var ; do x=55 ; done; 
      echo "$x"
    '
    

Process substitution is of course supported in ksh and zsh too. But since they run the last part of the pipeline in the main shell anyway, using it as a workaround isn't really necessary.

0
#!/bin/bash
set -x

# prepare test data.
mkdir -p ~/test_var_global
cd ~/test_var_global
echo "a"> core.1
echo "b"> core.2
echo "c"> core.3


var=0

coreFiles=$(find . -type f -name "core*")
while read -r file;
do
  # perform computations on $i
  ((var++))
done <<EOF
$coreFiles
EOF

echo $var

Result:
...
+ echo 3
3

it can work.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.