665

I notice that some scripts which I have acquired from others have the shebang #!/path/to/NAME while others (using the same tool, NAME) have the shebang #!/usr/bin/env NAME.

Both seem to work properly. In tutorials (on Python, for example), there seems to be a suggestion that the latter shebang is better. But, I don't quite understand why this is so.

I realize that, in order to use the latter shebang, NAME must be in the PATH whereas the first shebang does not have this restriction.

Also, it appears (to me) that the first would be the better shebang, since it specifies precisely where NAME is located. So, in this case, if there are multiple versions of NAME (e.g., /usr/bin/NAME, /usr/local/bin/NAME), the first case specifies which to use.

My question is why is the first shebang preferred to the second one?

2
  • 19
    See this answer...
    – jasonwryan
    Jan 21, 2012 at 1:10
  • 1
    @TheGeeko61: In my case I had something broken and some variables wasn't in env. So I suggest to use this shebang to verify if env is correctly loaded.
    – Micromega
    Jan 21, 2012 at 10:52

10 Answers 10

630

It isn't necessarily better.

The advantage of #!/usr/bin/env python is that it will use whatever python executable appears first in the user's $PATH.

The disadvantage of #!/usr/bin/env python is that it will use whatever python executable appears first in the user's $PATH.

That means that the script could behave differently depending on who runs it. For one user, it might use the /usr/bin/python that was installed with the OS. For another, it might use an experimental /home/phred/bin/python that doesn't quite work correctly.

And if python is only installed in /usr/local/bin, a user who doesn't have /usr/local/bin in $PATH won't even be able to run the script. (That's probably not too likely on modern systems, but it could easily happen for a more obscure interpreter.)

By specifying #!/usr/bin/python you specify exactly which interpreter will be used to run the script on a particular system.

Another potential problem is that the #!/usr/bin/env trick doesn't let you pass arguments to the intrepreter (other than the name of the script, which is passed implicitly). This usually isn't an issue, but it can be. Many Perl scripts are written with #!/usr/bin/perl -w, but use warnings; is the recommended replacement these days. Csh scripts should use #!/bin/csh -f -- but csh scripts are not recommended in the first place. But there could be other examples.

I have a number of Perl scripts in a personal source control system that I install when I set up an account on a new system. I use an installer script that modifies the #! line of each script as it installs it in my $HOME/bin. (I haven't had to use anything other than #!/usr/bin/perl lately; it goes back to times when Perl often wasn't installed by default.)

A minor point: the #!/usr/bin/env trick is arguably an abuse of the env command, which was originally intended (as the name implies) to invoke a command with an altered environment. Furthermore, some older systems (including SunOS 4, if I recall correctly) didn't have the env command in /usr/bin. Neither of these is likely to be a significant concern. env does work this way, a lot of scripts do use the #!/usr/bin/env trick, and OS providers aren't likely to do anything to break it. It might be an issue if you want your script to run on a really old system, but then you're likely to need to modify it anyway.

Another possible issue, (thanks to Sopalajo de Arrierez for pointing it out in comments) is that cron jobs run with a restricted environment. In particular, $PATH is typically something like /usr/bin:/bin. So if the directory containing the interpreter doesn't happen to be in one of those directories, even if it's in your default $PATH in a user shell, then the /usr/bin/env trick isn't going to work. You can specify the exact path, or you can add a line to your crontab to set $PATH (man 5 crontab for details).

Kevin's comment points out that Python's virtualenv creates a special case, where the environment installs a Python interpreter in a special directory that's inserted at the front of $PATH. For that particular environment (and perhaps others like it), the #!/usr/bin/env python trick (or python3?) is likely to be the best solution. (I haven't used virtualenv myself.)

40
  • 7
    If /usr/bin/perl is perl 5.8, $HOME/bin/perl is 5.12, and a script requiring 5.12 hardcodes /usr/bin/perl in the shebangs, it can be a major pain to run the script. I've rarely seen having /usr/bin/env perl grab perl from the PATH be a problem, but it is often very helpful. And it is much prettier than the exec hack! Jan 25, 2012 at 20:30
  • 14
    It's worth noting that, if you want to use a specific interpreter version, /usr/bin/env is still better. simply because there usually are multiple interpreter versions installed on your machine named perl5, perl5.12, perl5.10, python3.3, python3.32, etc. and if your app has only been tested on that specific version, you can still specify #!/usr/bin/env perl5.12 and be okay even if the user has it installed somewhere unusual. In my experience, 'python' is usually just a symlink to the system standard version (not necessarily the most recent version on the system).
    – root
    Jul 20, 2013 at 19:44
  • 10
    @root: For Perl, use v5.12; serves some of that purpose. And #!/usr/bin/env perl5.12 will fail if the system has Perl 5.14 but not 5.12. For Python 2 vs. 3, #!/usr/bin/python2 and #!/usr/bin/python3 are likely to work. Jul 20, 2013 at 21:44
  • 7
    If it doesn't work with /usr/bin/perl, I'll find out very quickly, and it's the system owner/administrator's responsibility to keep it up to date. If you want to run my script with your own perl, feel free to grab and modify a copy or invoke it via perl foo. (And you might consider the possibility that the 55 people who upvoted this answer also know a thing or two. It's certainly possible that you're right and they're all wrong, but that's not the way I'd bet.) Dec 14, 2013 at 22:03
  • 7
    This answer is not congruent with how Python in particular is commonly used. The Python executable you want to use is rarely /usr/bin/python. Typically, the one you want lives in a Virtualenv, and is at the front of the user's $PATH. In the rare situations where that is not the case, /usr/bin will usually be on the user's $PATH anyway.
    – Kevin
    Dec 26, 2015 at 23:31
129

Because /usr/bin/env can interpret your $PATH, which makes scripts more portable.

#!/usr/local/bin/python

Will only run your script if python is installed in /usr/local/bin.

#!/usr/bin/env python

Will interpret your $PATH, and find python in any directory in your $PATH.

So your script is more portable, and will work without modification on systems where python is installed as /usr/bin/python, or /usr/local/bin/python, or even custom directories (that have been added to $PATH), like /opt/local/bin/python.

Portability is the only reason using env is preferred to hard coded paths.

9
  • 27
    Custom directories for python executables are particularly common as virtualenv usage increases. Nov 22, 2013 at 0:01
  • 2
    What about #! python, why isn't that used?
    – kristianp
    May 14, 2018 at 23:54
  • 11
    #! python isn't used because you'd have to be in the same directory as the python binary, since the bareword python is interpreted as the complete path to the file. If you don't have a python binary in the current directory, you'll get an error like bash: ./script.py: python: bad interpreter: No such file or directory. It's the same as if you used #! /not/a/real/path/python May 16, 2018 at 16:14
  • 2
    @TimKennedy uname -a says: Linux silvius 5.6.0-2-amd64 #1 SMP Debian 5.6.14-1 (2020-05-23) x86_64 GNU/Linux, and I'm using zsh. zsh is the key: I have the same failure when running from bash, or with perl -e 'exec "/tmp/foo" or die $!'. TIL, zsh is doing way more work than it has to. Jun 1, 2020 at 7:15
  • 2
    @TimKennedy aha, from man zsh-misc: "If execution fails because the file is not in executable format, and the file is not a directory, it is assumed to be a shell script. /bin/sh is spawned to execute it. If the program is a file beginning with `#!', the remainder of the first line specifies an interpreter for the program. The shell will execute the specified interpreter on operating systems that do not handle this executable format in the kernel." So zsh has implemented its own shebang-line parsing in case it runs on a non-POSIX system, and its implementation differs from Linux kernel's. Jun 1, 2020 at 7:30
78

Objective Criteria/Requirements:

In determining whether to use an absolute or logical (/usr/bin/env) path to an interpreter in a shebang, there are two key considerations:

a) The interpreter can be found on the target system

b) The correct version of the interpreter can be found on the target system

If we AGREE that "b)" is desirable, we also agree that:

c) It's preferable our scripts fail rather than execute using an incorrect interpreter version and potentially achieve inconsistent results.

If we DON'T AGREE that "b)" matters, then any interpreter found will suffice.

Testing:

Since using a logical path — /usr/bin/env to the interpreter in the shebang — is the most extensible solution allowing the same script to execute successfully on target hosts with different paths to the same interpreter, we'll test it — using Python, due to its popularity — to determine whether it meets our criteria.

  1. Does /usr/bin/env live in a predictable, consistent location on POPULAR (not "every") operating systems? Yes:

    • RHEL 7.5
    • Ubuntu 18.04
    • Raspbian 10 ("Buster")
    • OSX 10.15.02
  2. Below Python script executed both inside and outside of virtual envelopes (Pipenv used) during tests:

    #!/usr/bin/env pythonX.x
    import sys
    print(sys.version)
    print('Hello, world!')
    
  3. The shebang in the script was varied by the Python version number desired (all installed on the same host):

    • #!/usr/bin/env python2
    • #!/usr/bin/env python2.7
    • #!/usr/bin/env python3
    • #!/usr/bin/env python3.5
    • #!/usr/bin/env python3.6
    • #!/usr/bin/env python3.7
  4. Expected results: that print(sys.version) = env pythonX.x.  Each time ./test1.py was executed using a different installed Python version, the correct version specified in the shebang was printed.

  5. Testing Notes:

    • Tests were exclusively limited to Python
    • Perl, like Python, MUST live in /usr/bin according to the FHS
    • I've not tested every possible combination on every possible number of Linuxy/Unixy Operating System and version of each Operating System.

Conclusion:

Although it's TRUE that #!/usr/bin/env python will use the first version of Python it matches in the user's Path, we can enforce an express preference by specifying a version number such as #!/usr/bin/env pythonX.x.  Indeed, developers don't care which interpreter is found "first"; all they care about is that their code is executed using the specified interpreter they know to be compatible with their code to ensure consistent results — wherever that may live in the filesystem...

In terms of portability/flexibility, using a logical/usr/bin/env — rather than absolute path not only meets requirements a), b) & c) from my testing with different versions of Python, but also has the benefit of fuzzy-logic finding the same version interpreter even if they live at different paths on different Operating Systems.  And although MOST distros respect the FHS, not all do.

So, where a script will FAIL if the binary lives in a different absolute path than specified in the shebang, the same script using a logical path SUCCEEDS as it keeps going until it finds a match, thereby offering greater reliability & extensibility across platforms.

7
  • 8
    Excellent analysis. We appreciate it.
    – TheGeeko61
    Feb 20, 2020 at 23:55
  • 4
    The question wasn't specific to Python. I don't think you can safely assume that other interpreters will be installed as, for example interpX and interpX.x. For example, the system I'm using at the moment has perl, perl5.26.3, and perl5.30.1. Another has perl and perl5.26.1. Neither has a perl5 command. Also, you didn't mention where the pythonX and pythonX.x interpreters on any of the systems you tested were installed. If they were all in /usr/bin then your experiment doesn't demonstrate any advantage of #!/usr/bin/env pythonX.x over #!/usr/bin/pythonX.x. Feb 21, 2020 at 2:55
  • 3
    Note that not every Linux system complies to the FHS, not by a long shot. And, paraphrasing the old saying, "Not all the world is Linux". Many Unix(y) systems have Python, Perl, even bash or tcsh as "optional, unsupported third party additions", presumably under /usr/local or /opt/*packagename* or some really exotic placements.
    – vonbrand
    Feb 23, 2020 at 4:01
  • 6
    UPVOTED for the formatting alone, never mind the very useful answer. Aug 26, 2020 at 13:30
  • 1
    @TeemuLeisti I try to save other technologists the effort of solving the same problems I bump into. So it's nice to hear somebody found my solution useful, especially when it's involved a fair amount of testing & documentation such as this one has. Most obliged for your kind words!
    – F1Linux
    Aug 26, 2020 at 18:22
58

Specifying the absolute path is more precise on a given system. The downside is that it's too precise. Suppose you realize that the system installation of Perl is too old for your scripts and you want to use your own instead: then you have to edit the scripts and change #!/usr/bin/perl to #!/home/myname/bin/perl. Worse, if you have Perl in /usr/bin on some machines, /usr/local/bin on others, and /home/myname/bin/perl on yet other machines, then you'd have to maintain three separate copies of the scripts and execute the appropriate one on each machine.

#!/usr/bin/env breaks if PATH is bad, but so does almost anything. Attempting to operate with a bad PATH is very rarely useful, and indicates that you know very little about the system the script is running on, so you can't rely on any absolute path anyway.

There are two programs whose location you can rely on on almost every unix variant: /bin/sh and /usr/bin/env. Some obscure and mostly retired Unix variants had /bin/env without having /usr/bin/env, but you're unlikely to encounter them. Modern systems have /usr/bin/env precisely because of its widespread use in shebangs. /usr/bin/env is something you can count on.

Apart from /bin/sh, the only time you should use an absolute path in a shebang is when your script isn't meant to be portable, so you can count on a known location for the interpreter. For example, a bash script that only works on Linux can safely use #!/bin/bash. A script that is only meant to be used in-house can rely on house interpreter location conventions.

#!/usr/bin/env does have downsides. It's more flexible than specifying an absolute path but still requires knowing the interpreter name. Occasionally you might want to run an interpreter that isn't in the $PATH, for example in a location relative to the script. In such cases, you can often make a polyglot script that can be interpreted both by the standard shell and by your desired interpreter. For example, to make a Python 2 script portable both to systems where python is Python 3 and python2 is Python 2, and to systems where python is Python 2 and python2 doesn't exist:

#!/bin/sh
''':'
if type python2 >/dev/null 2>/dev/null; then
  exec python2 "$0" "$@"
else
  exec python "$0" "$@"
fi
'''
# real Python script starts here
def …
28

Specifically for perl, using #!/usr/bin/env is a bad idea for two reasons.

First, it's not portable. On some obscure platforms env isn't in /usr/bin. Second, as Keith Thompson has noted, it can cause trouble with passing arguments on the shebang line. The maximally portable solution is this:

#!/bin/sh
exec perl -x "$0" "$@"
#!perl

For details on how it works, see 'perldoc perlrun' and what it says about the -x argument.

6
  • 1
    Exactly the anser I was looking for: how to write porable "shebang" for perl script that will allow additional aruments passed to perl (last line in your example accepts additional aruments). May 25, 2015 at 21:23
  • @AmokHuginnsson it is a bletcherous kludge.
    – vonbrand
    Feb 23, 2020 at 4:07
  • 1
    It is, but it's a bletcherous kludge that works. It's a blethcherous kludge that exists because less kludgey solutions don't work.
    – DrHyde
    Feb 24, 2020 at 21:59
  • What if sh is not in /bin? True story for google distroless containers.
    – LLlAMnYP
    Jul 28, 2020 at 16:17
  • 1
    Yeah, it also won't work if perl isn't in the path, or on an Amiga, or ... If you're using something weird like that then you're going to have to do some work yourself.
    – DrHyde
    Jul 29, 2020 at 11:10
20

The reason there is a distinction between the two is because of how scripts are executed.

Using /usr/bin/env (which, as mentioned in other answers, is not in /usr/bin on some OSes) is required because you can't just put an executable name after the #! - it must be an absolute path. This is because the #! mechanism works on a lower level than the shell. It's part of the kernel's binary loader. This can be tested. Put this in a file and mark it executable:

#!bash

echo 'foo'

You will find it prints an error like this when you attempt to run it:

Failed to execute process './test.sh'. Reason:
The file './test.sh' does not exist or could not be executed.

If a file is marked executable and begins with a #!, the kernel (which does not know about $PATH or the current directory: these are user-land concepts) will look for a file using an absolute path. Because using an absolute path is problematic (as mentioned in other answers), someone came up with a trick: You can run /usr/bin/env (which is almost always in that location) to run something using $PATH.

16

There are two more problems with using #!/usr/bin/env

  1. It doesn't solve the problem of specifying the full path to the interpreter, it just moves it to env.

    env is no more guaranteed to be in /usr/bin/env than bash is guaranteed to be in /bin/bash or python in /usr/bin/python.

  2. env overwrites ARGV[0] with the name of the interpreter (e..g bash or python).

    This prevents your script's name from appearing in, e.g., ps output (or changes how/where it appears) and makes it impossible to find it with, e.g, ps -C scriptname.sh

[update 2016-06-04]

And a third problem:

  1. Changing your PATH is more work than just editing the first line of a script, especially when scripting such edits is trivial. e.g:

    printf "%s\n" 1 i '#!'$(type -P python2) . w | ed foo.py

    Appending or pre-pending a directory to $PATH is fairly easy (although you still have to edit a file to make it permanent - your ~/.profile or whatever - and it's far from easy to script THAT edit because the PATH could be set anywhere in the script, not in the first line).

    Changing the order of PATH directories is significantly more difficult....and far harder than just editing the #! line.

    And you still have all the other problems that using #!/usr/bin/env gives you.

    @jlliagre suggests in a comment that #!/usr/bin/env is useful for testing your script with multiple interpreter versions, "by only changing their PATH / PATH order"

    If you need to do that, it's much easier to just have several #! lines at the top of your script (they're just comments anywhere but the very first line) and cut/copy-and-paste the one you want to use now to the first line.

    In vi, that would be as simple as moving the cursor to the #! line you want, then typing dd1GP or Y1GP. Even with an editor as trivial as nano, it would take seconds to use the mouse to copy and paste.


Overall, the advantages of using #!/usr/bin/env are minimal at best, and certainly don't even come close to outweighing the disadvantages. Even the "convenience" advantage is largely illusory.

IMO, it's a silly idea promoted by a particular kind of programmer who thinks that operating systems are not something to be worked with, they are a problem to be worked around (or ignored at best).

PS: here's a simple script to change the interpreter of multiple files at once.

change-shebang.sh:

#!/bin/bash

interpreter="$1"
shift

if [ -z "$(type -P $interpreter)" ] ; then
  echo "Error: '$interpreter' is not executable." >&2
  exit 1
fi

if [ ! -d "$interpreter" ] && [ -x "$interpreter" ] ; then
  shebang='#!'"$(realpath -e $interpreter)" || exit 1
else
  shebang='#!'"$(type -P $interpreter)"
fi

for f in "$@" ; do
  printf "%s\n" 1 i "$shebang" . w | ed "$f"
done

Run it as, e.g., change-shebang.sh python2.7 *.py or change-shebang.sh $HOME/bin/my-experimental-ruby *.rb

14
  • 6
    Nobody else here has mentioned the ARGV[0] problem. And nobody has mentioned the /path/to/env issue in a form that directly addresses one of the arguments for using it (i.e. that bash or perl might be in an unexpected location).
    – cas
    Oct 25, 2015 at 7:55
  • 3
    The interpreter path issue is easily fixed by a sysadmin with symlinks, or by the user editing their script. It's certainly not a significant enough problem to encourage people to put up with the all of the other issues caused by using env on the shebang line that are mentioned here and on other questions. That's promoting a bad solution in the same kind of way that encouraging csh scripting is promoting a bad solution: it kind of works but there are much better alternatives.
    – cas
    Oct 25, 2015 at 8:37
  • 2
    Not everybody is a sysadmin on their machine. The env solution is aimed at helping non operating system specialists to copy/paste or download scripts that have a good chance to work as is. It is also one of the methods that allow more advanced users to experiment with multiple unchanged scripts to be executed by custom versions of the target interpreter by only changing their PATH / PATH order (sorry about that too long sentence...)
    – jlliagre
    Oct 25, 2015 at 10:38
  • 3
    non-sysadmins can ask their sysadmin to do it. or they can simply edit the script and change the #! line.
    – cas
    Oct 25, 2015 at 10:51
  • 6
    That is precisely what env is helping to avoid: depending on a sysadmin or OS specific technical knowledge unrelated to python or whatever.
    – jlliagre
    Oct 25, 2015 at 17:27
12

Adding another example here:

Using env is also useful when you want to share scripts between multiple rvm environments for example.

Running this on the cmd line, shows which ruby version will be used when #!/usr/bin/env ruby is used inside a script:

env ruby --version

Therefore, when you use env, you can use different ruby versions through rvm, without changing your scripts.

2
  • 1
    // , Excellent idea. The whole point of multiple interpreters is NOT to break the code, or to have code depend upon that specific interpreter. Jan 15, 2016 at 20:52
  • Or maybe you could use different interpreter name if you change the language? If you put env ruby at the front of your script and you actually require some specific version, you're doing it backwards. The env ruby should mean any version of ruby is okay. Both python and ruby have this same problem where the language syntax or features is different but different people pretend to use the same language and use identical shebang. Apr 14, 2022 at 11:33
7

If you're writing purely for yourself or for your job and the location of the interpreter you're calling is always in the same place, by all means use the direct path. In all other cases use #!/usr/bin/env.

Here's why: in your situation the python interpreter was in same place regardless of which syntax you used but for many people it could have installed in a different place. Though most major programming interpreters are located in /usr/bin/ a lot of newer software is defaulting to /usr/local/bin/.

I would even argue to always use #!/usr/bin/env because if you have multiple versions of the same interpreter installed and you don't know what your shell will default to, then you should probably fix that.

2
  • 8
    "In all other cases use #!/usr/bin/env" is too strong. // As @KeithThompson and others point out, #!/usr/bin/env means the script can behave differently depending on who/how run. Sometimes this different behavior can be a security bug. // For example, NEVER use "#!/usr/bin/env python" for a setuid or setgid script, because the user invoking the script can place malware in a file called "python" in PATH. Whether setuid/gid scripts are ever a good idea is a different issue - but certainly, no setuid/gid executable should ever trust the user provided environment.
    – Krazy Glew
    Jun 3, 2016 at 18:07
  • 3
    @KrazyGlew, you can't setuid a script on Linux. You do it trough an executable. And when writing this executable, it is good practice, and widely done, to clear the environment variables. Some environment variable are also purposefully ignored.
    – MayeulC
    Oct 7, 2018 at 11:28
7

For portability and compatiblity reasons it is better to use

#!/usr/bin/env bash

instead of

#!/usr/bin/bash

There are multple possiblities, where a binary may be located on an Linux/Unix system. Check the hier(7) manpage for a detailed description of the file system hierarchy.

FreeBSD for example installs all software, which is not part of the base system, in /usr/local/. Since the bash is not part of the base system, the bash binary is installed at /usr/local/bin/bash.

When you want a portable bash/csh/perl/whatever script, which runs under most Linux Distributions and FreeBSD, you should use #!/usr/bin/env.

Also take note, that most Linux installations have also (hard)linked the env binary to /bin/env or softlinked /usr/bin into /bin which should not be used in the shebang. So don't use #!/bin/env.

1
  • 1
    To paraphrase, "not all the world's a Linux". And not all Linux systems abide by hier(7) either. Most non-Linux systems just don't have bash, or it is a "unsupported, unofficial addon" buried somewhere obscure.
    – vonbrand
    Feb 23, 2020 at 4:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .