9

If I run this bash command and prefix the statement, so that the variable fruit should exist, but only for the duration of this command:

$ fruit=apple echo $fruit

$

The result is an empty line. why?

To quote a comment from wildcard on this question:

parameter expansion is done by the shell, and the "fruit" variable is not a shell variable; it's only an environment variable within the environment of the "echo" command

An environment variable is still a variable, so surely this should still be available to the echo command?

marked as duplicate by muru, Stephen Kitt, l0b0, G-Man, Kusalananda May 30 '17 at 9:51

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    fruit=apple; echo $fruit – jasonwryan May 30 '17 at 1:40
  • 1
    @jasonwryan that was a fast response! but that will set the variable permanently – the_velour_fog May 30 '17 at 1:41
  • 2
    @muru This question is not a duplicate of the IFS one. That's a different issue here. – jlliagre May 30 '17 at 2:12
  • 1
    @muru The issues are only loosely similar but the issues root causes are unrelated. The string $IFS appears nowhere in the other question while here, the main issue is $fruit appears in the command line. – jlliagre May 30 '17 at 2:28
  • 1
    @muru The other question is related to word splitting while that one is about variable expansion. While both are part of the various tasks the shell parser is performing, I still believe the questions are different. No big deal anyway. – jlliagre May 30 '17 at 2:47
9

To understand this properly, let's first differentiate shell variables from environment variables.

Environment variables are a property that ALL processes have, whether they utilize them internally or not. Even sleep 10 when it's running has environment variables. Just as all processes have a PID (process identifier), a current working directory (cwd), a PPID (parent PID), an argument list (even if empty)—and so on. So too do all processes have what's called an "environment," which is inherited from the parent process when it forks.

From a utility author perspective (someone who writes code in C), processes have the ability to set, unset or change environment variables. However, from a script author perspective, most tools do not offer that facility to their users. Instead, you use your shell to alter the process environment which is then inherited when the command (external binary) you called is executed. (The environment of your shell itself can be modified and the modification inherited, or you can direct your shell to make the modification after forking but prior to executing the command you called. Either way, the environment is inherited. We'll look at both approaches.)

Shell variables are another thing. Although within the shell they behave in the same way, the difference is that mere "shell variables" do not alter or influence the behavior of commands that you call from your shell. In proper terminology the distinction would actually be stated a little bit differently; exported shell variables will become part of the environment of tools which you call, whereas shell variables which are not exported will not. However, I find it more helpful for communication to refer to shell variables which are not exported as "shell variables" and shell variables which are exported as "environment variables" because they are environment variables from the perspective of processes forked from the shell.


That's a lot of text. Let's look at some examples and describe what's happening:

$ somevar=myfile
$ ls -l "$somevar"
-rw-r--r--  1 Myname  staff  0 May 29 19:12 myfile
$ 

In this example, somevar is just a shell variable, nothing special about it. Shell parameter expansion (see LESS='+/Parameter Expansion' man bash) occurs before the ls executable is actually loaded ("exec"ed) and the ls command (process) never even sees the string "dollar sign s-o-m-e-v-a-r". It only sees the string "m-y-f-i-l-e", interprets that as the path to a file in the current working directory, and gets and prints information about it.

If we run export somevar before the ls command, then the fact that somevar=myfile will appear in the environment of the ls process, but that won't affect anything because the ls command won't do anything with this variable. To see an effect of an environment variable, we have to choose an environment variable that the process we're calling will actually check for and do something with.


bc: Basic Calculator

There may be a better example, but this is one I came up with that's not too complicated. First you should know that bc is basic calculator, and processes and computes mathematical expressions. (After processing the contents of any input files, it processes its standard input. I'm not going to use its standard input for my examples; I'll just press Ctrl-D, which will not show in the text snippets below. Also I'm using -q to suppress an introductory message on every invocation.)

The environment variable I will illustrate is described in man bc:

   BC_ENV_ARGS
      This is another mechanism to get arguments to bc.  The format is
      the  same  as  the  command line arguments.  These arguments are
      processed first, so any files listed in  the  environment  argu-
      ments  are  processed  before  any  command line argument files.
      This allows the user to set up "standard" options and  files  to
      be  processed at every invocation of bc.  The files in the envi-
      ronment variables would typically contain  function  definitions
      for functions the user wants defined every time bc is run.

Here goes:

$ cat file1
5*5
$ bc -q file1
25
$ cat file2
6*7
8+9+10
$ bc -q file2
42
27
$ bc -q file1 file2
25
42
27
$

This is just to show how bc works. In each of these instances I had to press Ctrl-D to signal "end of input" to bc.

Now, let's pass an environment variable directly to bc:

$ BC_ENV_ARGS=file1 bc -q file2
25
42
27
$ echo "$BC_ENV_ARGS"

$ bc -q file2
42
27
$

Notice that what we put in that variable is not visible later from an echo command. By putting the assignment as part of the same command (with no semicolon, either), we put that variable assignment as part of the environment of bc—it left the shell itself that we are running unaffected.

Now let's set BC_ENV_ARGS as a shell variable:

$ BC_ENV_ARGS=file1
$ echo "$BC_ENV_ARGS"
file1
$ bc -q file2
42
27
$

Here you can see that our echo command can see the contents, but it's not part of the environment of bc so bc can't do anything special with it.

Of course if we put the variable itself in bc's argument list we would see something:

$ bc -q "$BC_ENV_ARGS"
25
$ 

But here, it is the shell expanding the variable, and then file1 is what actually appears in bc's argument list. So this is still using it as a shell variable, not an environment variable.

Now let's "export" this variable so it's both a shell variable AND an environment variable:

$ export BC_ENV_ARGS
$ echo "$BC_ENV_ARGS"
file1
$ bc -q file2
25
42
27
$

And here you can see that file1 is processed before file2, even though not mentioned on the command line here. It's part of the shell's environment and becomes part of bc's environment when you run that process, so the value of this environment variable is inherited and affects how bc operates.

We can still override this on a per-command basis, even override it to an empty value:

$ BC_ENV_ARGS= bc -q file2
42
27
$ echo "$BC_ENV_ARGS"
file1
$ bc -q file2
25 
42
27
$ 

But as you can see, the variable remains set and exported in our shell, visible both to the shell itself and to any later bc commands that don't override the value. It will remain this way unless we "unexport" or "unset" it. I'll do the latter:

$ unset BC_ENV_ARGS
$ echo "$BC_ENV_ARGS"

$ bc -q file2
42
27
$ 

Another example, involving spawning another shell:

Type the following commands one after another into your shell and consider the results. See if you can predict the results before running them.

# fruit is not set
echo "$fruit"
sh -c 'echo "$fruit"'
# fruit is set as a shell variable in the current shell only
fruit=apple
echo "$fruit"
sh -c 'echo "$fruit"'
sh -c "echo $fruit" ### NOT advised for use in scripts, for illustration only
# fruit is exported, so it's accessible in current AND new processes
export fruit
echo "$fruit"
sh -c 'echo "$fruit"'
echo '$fruit' ### I threw this in to make sure you're not confused on quoting
# fruit is unset again
unset fruit
echo "$fruit"
sh -c 'echo "$fruit"'
# setting fruit directly in environment of single command but NOT in current shell
fruit=apple sh -c 'echo "$fruit"'
echo "$fruit"
fruit=apple echo "$fruit"
# showing current shell is unaffected by directly setting env of single command
fruit=cherry
echo "$fruit"
fruit=apricot sh -c 'echo "$fruit"'
echo "$fruit"
sh -c 'echo "$fruit"'

And one last one for extra trickiness: Can you predict the output of the following commands run in sequence? :)

fruit=banana
fruit=orange sh -c 'fruit=lemon echo "$fruit"; echo "$fruit"; export fruit=peach'
echo "$fruit"

Please mention in the comments any clarifications desired; I'm sure this could use a few. But it should hopefully be helpful even just as it is.

  • thanks, if Im understnding right, you are saying the processes can distinguish between environment and the shell variables and bc "selects" only shell vars. agreed. but then when you say To see an effect of an environment variable, we have to choose an environment variable that the process we're calling will actually check for and do something with well echo will use both environment and shell variables. so there must be something else happening before echo is run. if you havent already, I recommend reading Gilles answer I linked to above – the_velour_fog May 30 '17 at 3:18
  • @the_velour_fog not quite. Shell variables (that have not been exported) are never seen by the process you call, at all. If they're specified with $ in the command line, they are expanded by the shell as part of parsing the command you typed and the expansion (value) is passed as arguments to the command you call. So the command (whether echo or bc or whatever) only sees the expansion. However, if you set the shell variable fruit=apple and then convert it to an environment variable (one that will be visible to forked processes), it won't affect commands you call unless (cont'd) – Wildcard May 30 '17 at 3:51
  • they look for the variable fruit in their environment. I'll add an example with the sh command to show this. – Wildcard May 30 '17 at 3:52
9

The issue is the current shell is expanding the variable too early; it is not set in its context so the echo command doesn't get any argument, i.e. the commands ends up being:

$ fruit=apple echo

Here is a workaround where the variable doesn't get expanded too early because of the single quotes:

$ fruit=apple sh -c 'echo $fruit'

Alternatively, you can also use a one line shell script which demonstrates the fruit variable is correctly passed to the executed command:

$ cat /tmp/echof
echo $fruit
$ /tmp/echof

$ fruit=apple /tmp/echof
apple
$ echo $fruit

$

A few comments as this question has sparkled some unexpected controversy and discussions:

  • The fact the variable fruit is already exported or not doesn't affect the behavior, what matters is what the variable value is at the precise moment the shell is expanding it.
$ export fruit=banana
$ fruit=apple echo $fruit
banana
  • The fact the echo command is a builtin one doesn't affect the OP issue. However, there are cases where using builtins or shell functions with this syntax have unexpected side effects, e.g.:
$ export fruit=banana
$ fruit=apple eval 'echo $fruit'
apple
$ echo $fruit
apple
  • While there is a similarity between the question asked here and that one, this is not exactly the same issue. With that other question, the temporary IFS variable value is not yet available when the shell word split another variable $var while here, the temporary fruit variable value is not yet available when the shell expands the same variable.

  • There is also that other question where the OP is asking about the significance of the syntax used and more precisely is asking "why does this work?". Here the OP is aware of the significance but reports an unexpected behavior and asks about its cause, i.e. "why doesn't this work?". Ok, after reading more closely the poor screenshot posted on the other question, the same situation is indeed described there (BAZ=jake echo $BAZ) so yes, after all this is a duplicate

  • but with other commands like IFS='' read the $IFS variable will be "set in its context". Whats not clear to me is why IFS is "set" but why $fruit is not? – the_velour_fog May 30 '17 at 1:57
  • fruit is set too but echo has no chance to know you want to display it. – jlliagre May 30 '17 at 1:59
  • 1
    No, this particular issue is unrelated to the fact echo is a builtin. It would happen with a standard command too. – jlliagre May 30 '17 at 2:10
  • 2
    So in other words, the shell parses it into "set this variable, run this command, with the value of fruit expanded", and then evaluates the "set this variable" and "run this command" parts. – tripleee May 30 '17 at 4:15
  • 1
    @the_velour_fog, the difference between echo and read is that one takes what it gets from the command line, the other uses IFS internally. If you had a command print_fruit that used the variable internally, then fruit=apple print_fruit would output apple. (You can emulate print_fruit with a shell script containing echo "$fruit") – ilkkachu May 30 '17 at 5:49
3

Because expansions on the command line take place before variable assignments. The standard says so:

When a given simple command is required to be executed, the following ... shall all be performed:

  1. The words that are recognized as variable assignments [...] are saved for processing in steps 3 and 4.

  2. The words that are not variable assignments or redirections shall be expanded. [...]

  3. Redirections shall be performed as described in Redirection.

  4. Each variable assignment shall be expanded [...] prior to assigning the value.

Note the order: in the first step, assignments are only saved, then the other words are expanded, and only in the end do variable assignments take place.

Of course, it's highly likely that standard says so only because it's always been like that, and they just codified the existing behaviour. It doesn't say anything about the history or the reasoning behind it. The shell has to identify words that look like assignments at some point (if only to not take them as part of the command), so I suppose it could work in the other order instead, assign variables first, then expand anything on the command line. (or just do it left-to-right...)

  • thanks, if expansions take place before assignments how does the IFS=. read work? if that spec was applicable, wouldnt that mean the IFS be taking effect (on word splitting) before the IFS=. got assigned? – the_velour_fog May 30 '17 at 5:51
  • 1
    @the_velour_fog, there are no expansions in IFS=. read. An expansion means $variablename. – Wildcard May 30 '17 at 5:54
  • 1
    @the_velour_fog, read only uses IFS after the command line is processed, it acts like any command (though it implicitly sees even an unexported IFS). If you did something like vars="a b c"; IFS=. read $vars <<< "1.2.3"; echo $b (with the default value of IFS) you'd get 2, since $vars was word-split with the original IFS, but read used the one explicitly given to it. – ilkkachu May 30 '17 at 6:03

Not the answer you're looking for? Browse other questions tagged or ask your own question.