8

I am setting certain environment variables for the command pyspark to work. When I set the variables in /etc/environment and source it, it doesn't work. However, when I set them in command line they do work but ofcourse only for this session. My intent is to set them globally so that even if I re-open the session I can just type pyspark

Setting in /etc/environment

[root@localhost ~]# more /etc/environment
[root@localhost ~]# echo "export SPARK_HOME=/srv/spark" >> /etc/environment
[root@localhost ~]# echo "export PATH="$SPARK_HOME"/bin:"$PATH >> /etc/environment
[root@localhost ~]# echo "export JAVA_HOME=/usr/lib/jvm/jre-1.7.0-openjdk" >> /etc/environment
[root@localhost ~]# source /etc/environment
[root@localhost ~]# pyspark --version
-bash: pyspark: command not found

Setting on command line

[root@localhost ~]# export SPARK_HOME=/srv/spark
[root@localhost ~]# export PATH=$SPARK_HOME/bin:$PATH
[root@localhost ~]# export JAVA_HOME=/usr/lib/jvm/jre-1.7.0-openjdk
[root@localhost ~]# pyspark --version
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.1
      /_/

Type --help for more information.
0

2 Answers 2

15

Put the export SPARK_HOME=... etc. commands in the startup files of your shell. With bash, that would be either ~/.profile or ~/.bash_profile.


On Linux, /etc/environment is usually read by pam_env.so during login, and it doesn't support expanding existing variables, so setting PATH=$PATH:/something will result in the literal string $PATH to appear in your PATH. This isn't what you want. (See e.g. this and this, also for fun this.)

Also, setting PATH in /etc/environment might not work, since the global startup scripts for the shell might rewrite them. (They do on Debian by default, on the old CentOS I have handy, the startup scripts only seem to prepend to PATH).

If your system doesn't use pam_env.so, but you only source the script by hand, then these considerations don't matter, of course. But it looks like it's widely used by at least a couple of Linux distributions, so it might be a good idea to use another filename.


(Because this is completely opposite to what the other answers said, I tested it on an old CentOS.)

I put the following in /etc/environment:

export FOO1=bar
export FOO2=foo:$FOO

After logging in again, set | grep FOO shows:

FOO1=bar
FOO2='foo:$FOO'
2
  • (I didn't have a current CentOS handy to test, so do point me wrong if I am.)
    – ilkkachu
    Jul 29, 2016 at 19:54
  • 1
    Nice twist. Shows that knowing a bit can lead to answers, but knowing the full picture will lead to a solution.
    – Kusalananda
    Jul 29, 2016 at 19:55
1

This line doesn't do what you think:

echo "export PATH="$SPARK_HOME"/bin:"$PATH >> /etc/environment

The problem is that $SPARK_HOME is being evaluated during the echo (as is $PATH; you probably don't want your current PATH in that file).

If you cat /etc/environment you'll see the line doesn't look right.

Instead, use

echo 'export PATH="$SPARK_HOME/bin:$PATH"' >> /etc/environment

This will use the values of $SPARK_HOME and $PATH in place when you source the file.

2
  • 1
    No, your proposal has exactly the same bug as the original. You're writing a shell program to /etc/environment, but that's invalid syntax. Jul 29, 2016 at 22:57
  • There's two different bugs. The first bug is that what was being written out to /etc/environment was being evaluated at the wrong point. That's what I fixed. With that fix the OP can source the file as per his example and it'll work. There's a second logical bug if he wants that file to be used in pam_env in that $VAR syntax isn't understood. Jul 29, 2016 at 23:07

Not the answer you're looking for? Browse other questions tagged or ask your own question.