21

I have a script that I want to be able to run in two machines. These two machines get copies of the script from the same git repository. The script needs to run with the right interpreter (e.g. zsh).

Unfortunately, both env and zsh live in different locations in the local and remote machines:

Remote machine

$ which env
/bin/env

$ which zsh
/some/long/path/to/the/right/zsh

Local machine

$ which env
/usr/bin/env

$which zsh
/usr/local/bin/zsh

How can I set up the shebang so that running the script as /path/to/script.sh always uses the Zsh available in PATH?

1
  • 9
    Are you sure env isn't in both /bin and /usr/bin? Try which -a env to confirm. Commented Apr 28, 2014 at 7:27

4 Answers 4

23

You cannot solve this through shebang directly, since shebang is purely static. What you could do is having some »least common multiplier« (from a shell perspective) in the shebang and re-execute your script with the right shell, if this LCM isn't zsh. In other words: Have your script executed by a shell found on all systems, test for a zsh-only feature and if the test turns out false, have the script exec with zsh, where the test will succeed and you just continue.

One unique feature in zsh, for example, is the presence of the $ZSH_VERSION variable:

#!/bin/sh -

[ -z "$ZSH_VERSION" ] && exec zsh - "$0" ${1+"$@"}

# zsh-specific stuff following here
echo "$ZSH_VERSION"

In this simple case, the script is first executed by /bin/sh (all post-80s Unix-like systems understand #! and have a /bin/sh , either Bourne or POSIX but our syntax is compatible to both). If $ZSH_VERSION is not set, the script exec's itself through zsh. If $ZSH_VERSION is set (resp. the script already is run through zsh), the test is simply skipped. Voilà.

This only fails if zsh isn't in the $PATH at all.

Edit: To make sure, you only exec a zsh in the usual places, you could use something like

for sh in /bin/zsh \
          /usr/bin/zsh \
          /usr/local/bin/zsh; do
    [ -x "$sh" ] && exec "$sh" - "$0" ${1+"$@"}
done

This could save you from accidentally exec'ing something in your $PATH which is not the zsh you're expecting.

6
  • I upvoted this for elegance but it does, in principle, have security/compatibility issues, if the first zsh in the $PATH is not the one you expect.
    – Ryan Reich
    Commented Apr 28, 2014 at 5:06
  • Tried to address it. Question is, whether you can always be sure whether a zsh binary in the standard locations really is a zsh. Commented Apr 28, 2014 at 5:57
  • You can dynamically path the !bang line. You can also ask zsh itself where it is with zsh -c 'whence zsh'. More simply you can just command -v zsh. See my answer for how to dynamically path the #!bang.
    – mikeserv
    Commented Apr 28, 2014 at 13:16
  • 1
    Calling the zsh binary from $PATH to get the path of the zsh binary wouldn't quite address the problem @RyanReich pointed out, would it? :) Commented Apr 28, 2014 at 13:20
  • Not if you exec zsh itself, no, I guess not. But if you embed the resulting string in your hash bang and then exec your own script you at least know what you're getting. Still, it would make for a simpler test than for looping it.
    – mikeserv
    Commented Apr 28, 2014 at 13:49
7

For years I've used something similar to deal with the various locations of Bash on systems that I needed my scripts to run.

Bash/Zsh/etc.

#!/bin/sh

# Determines which OS and then reruns this script with approp. shell interp.
LIN_BASH="/bin/sh";
SOL_BASH="/packages/utilities/bin/sun5/bash";

OS_TYPE=`uname -s`;

if [ $OS_TYPE = "SunOS" ]; then
  $SOL_BASH -c "`sed -n '/\#\#\# BEGIN/,$p' $0`" $0 $*;
elif [ $OS_TYPE = "Linux" ]; then
  $LIN_BASH -c "`sed -n '/\#\#\# BEGIN/,$p' $0`" $0 $*;
else
  echo "UNKNOWN OS_TYPE, $OS_TYPE";
  exit 1;
fi
exit 0;

### BEGIN

...script goes here...

The above can be easily adapted for a variety of interpreters. The key piece is that this script initially runs as Bourne shell. It then recursively calls itself a second time, but parses everything out above the comment ### BEGIN using sed.

Perl

Here's a similar trick for Perl:

#!/bin/sh

LIN_PERL="/usr/bin/perl";
SOL_PERL="/packages/perl/bin/perl";

OS_TYPE=`uname -s`;

if [ $OS_TYPE = "SunOS" ]; then
  eval 'exec $SOL_PERL -x -S $0 ${1+"$@"}';
elif [ $OS_TYPE = "Linux" ]; then
  eval 'exec $LIN_PERL -x -S $0 ${1+"$@"}';
else
  echo "$OS_TYPE: UNSUPORRTED OS/PLATFORM";
  exit 0;
fi
exit 0;

#!perl

...perl script goes here...

This method makes use of Perl's ability when given an file to run will parse said file skipping all the lines that are before the line #! perl.

1
  • A number of issues there: missing quotes, use of $* instead of "$@", useless use of eval, exit status not reported (you didn't use exec for the first one), missing -/--, error messages not on stderr, 0 exit status for error conditions, using /bin/sh for LIN_BASH, useless semicolon (cosmetic), using all uppercase for non-env variables. uname -s is like uname (uname is for Unix name). You forgot to mention that the skipping is triggered by the -x option to perl. Commented May 20, 2014 at 14:11
4

NOTE: @jw013 makes the following unsupported objection in the comments below:

The downvote is because self-modifying code is generally considered bad practice. Back in the old days of tiny assembly programs it was a clever way to reduce conditional branches and improve performance, but nowadays the security risks outweigh the advantages. Your approach would not work if the user who ran the script did not have write privileges on the script.

I answered his security objections by pointing out that any special permissions are only required once per install/update action in order to install/update the self-installing script - which I would personally call pretty secure. I also pointed him to a man sh reference to achieving similar goals by similar means. I did not, at the time, bother to point out that whatever security flaws or otherwise generally unadvised practices that may or may not be represented in my answer, they were more likely rooted in the question itself than they were in my answer to it:

How can I set up the shebang so that running the script as /path/to/script.sh always uses the Zsh available in PATH?

Not satisfied, @jw013 continued to object by furthering his as yet unsupported argument with at least a couple of erroneous statements:

You use a single file, not two files. The [man sh referenced] package has one file modify another file. You have a file modifying itself. There is a distinct difference between these two cases. A file that takes input and produces output is fine. An executable file that changes itself as it runs is generally a bad idea. The example you pointed to does not do that.

In the first place:

THE ONLY EXECUTABLE CODE IN ANY EXECUTABLE SHELL SCRIPT IS the #! ITSELF

(though even #! is officially unspecified)

{   cat >|./file 
    chmod +x ./file 
    ./file
} <<-\FILE
    #!/usr/bin/sh
    {   ${l=lsof -p} $$
        echo "$l \$$" | sh
    } | grep \
        "COMMAND\|^..*sh\| [0-9]*[wru] "
#END
FILE

##OUTPUT

COMMAND  PID     USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
file    8900 mikeserv  txt    REG   0,33   774976  2148676 /usr/bin/bash
file    8900 mikeserv  mem    REG   0,30           2148676 /usr/bin/bash (path dev=0,33)
file    8900 mikeserv    0r   REG   0,35      108 15496912 /tmp/zshUTTARQ (deleted)
file    8900 mikeserv    1u   CHR  136,2      0t0        5 /dev/pts/2
file    8900 mikeserv    2u   CHR  136,2      0t0        5 /dev/pts/2
file    8900 mikeserv  255r   REG   0,33      108  2134129 /home/mikeserv/file
COMMAND  PID     USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
sh      8906 mikeserv  txt    REG   0,33   774976  2148676 /usr/bin/bash
sh      8906 mikeserv  mem    REG   0,30           2148676 /usr/bin/bash (path dev=0,33)
sh      8906 mikeserv    0r  FIFO    0,8      0t0 15500515 pipe
sh      8906 mikeserv    1w  FIFO    0,8      0t0 15500514 pipe
sh      8906 mikeserv    2u   CHR  136,2      0t0        5 /dev/pts/2

{    sed -i \
         '1c#!/home/mikeserv/file' ./file 
     ./file 
     sh -c './file ; echo'
     grep '#!' ./file
}

##OUTPUT
zsh: too many levels of symbolic links: ./file
sh: ./file: /home/mikeserv/file: bad interpreter: Too many levels of symbolic links

#!/home/mikeserv/file

A shell script is just a text file - in order for it to have any effect at all it must be read by another executable file, its instructions then interpreted by that other executable file, before finally the other executable file then executes its interpretation of the shell script. It is not possible for the execution of a shell script file to involve fewer than two files. There is a possible exception in zsh's own compiler, but with this I have little experience and it is in no way represented here.

A shell script's hashbang must point to its intended interpreter or be discarded as irrelevant.

THE SHELL'S TOKEN RECOGNITION/EXECUTION BEHAVIOR IS STANDARDS-DEFINED

The shell has two basic modes of parsing and interpreting its input: either its current input is defining a <<here_document or it is defining a { ( command |&&|| list ) ; } & - in other words, the shell either interprets a token as a delimiter for a command it should execute once it has read it in or as instructions to create a file and map it to a file descriptor for another command. That's it.

When interpreting commands to execute the shell delimits tokens on a set of reserved words. When the shell encounters an opening token it must continue to read in a command list until the list is either delimited by a closing token such as a newline - when applicable - or the closing token like }) for ({ before execution.

The shell distinguishes between a simple command and a compound command. The compound command is the set of commands that must be read in before execution, but the shell does not perform $expansion on any of its constituent simple commands until it singly executes each one.

So, in the following example, the ;semicolon reserved words delimit individual simple commands whereas the non-escaped \newline character delimits between the two compound commands:

{   cat >|./file
    chmod +x ./file
    ./file
} <<-\FILE
        #!/usr/bin/sh
        echo "simple command ${sc=1}" ;\
                : > $0 ;\
                echo "simple command $((sc+2))" ;\
                sh -c "./file && echo hooray"
        sh -c "./file && echo hooray"
#END
FILE

##OUTPUT

simple command 1
simple command 3
hooray

That is a simplification of the guidelines. It gets much more complicated when you consider shell-builtins, subshells, current environment and etc, but, for my purposes here, it is enough.

And speaking of built-ins and command-lists, a function() { declaration ; } is merely a means of assigning a compound command to a simple command. The shell must not perform any $expansions on the declaration statement itself - to include <<redirections> - but must instead store the definition as a single, literal string and execute it as a special shell built-in when called upon.

So a shell function declared in an executable shell script is stored in the interpreting shell's memory in its literal string form - unexpanded to include appended here-documents as input - and executed independently of its source file every time it is called as a shell built-in for as long as the shell's current environment lasts.

A <<HERE-DOCUMENT IS AN INLINE FILE

The redirection operators << and <<- both allow redirection of lines contained in a shell input file, known as a here-document, to the input of a command.

The here-document shall be treated as a single word that begins after the next \newline and continues until there is a line containing only the delimiter and a \newline, with no [:blank:]s in between. Then the next here-document starts, if there is one. The format is as follows:

[n]<<word
    here-document 
delimiter

...where the optional n represents the file descriptor number. If the number is omitted, the here-document refers to standard input (file descriptor 0).

for shell in dash zsh bash sh ; do sudo $shell -c '
        {   readlink /proc/self/fd/3
            cat <&3
        } 3<<-FILE
            $0

        FILE
' ; done

#OUTPUT

pipe:[16582351]
dash

/tmp/zshqs0lKX (deleted)
zsh

/tmp/sh-thd-955082504 (deleted)
bash

/tmp/sh-thd-955082612 (deleted)
sh

You see? For every shell above the shell creates a file and maps it to a file descriptor. In zsh, (ba)sh the shell creates a regular file in /tmp, dumps output, maps it to a descriptor, then deletes the /tmp file so the kernel's copy of the descriptor is all that remains. dash avoids all of that nonsense and simply drops its output processing into an anonymous |pipe file aimed at the redirect << target.

This makes dash's:

cmd <<HEREDOC
    $(cmd)
HEREDOC

functionally equivalent to bash's:

cmd <(cmd)

while dash's implementation is at least POSIXly portable.

WHICH MAKES SEVERAL FILES

So in the answer below when I do:

{    cat >|./file
     chmod +x ./file
     ./file
} <<\FILE
#!/usr/bin/sh
_fn() { printf '#!' ; command -v zsh ; cat 
} <<SCRIPT >$0
    [SCRIPT BODY]
SCRIPT    

_fn ; exec $0
FILE

The following happens:

  1. I first cat the contents of whatever file the shell created for FILE into ./file, make it executable, then execute it.

  2. The kernel interprets the #! and calls /usr/bin/sh with a <read file descriptor assigned to ./file.

  3. sh maps a string into memory consisting of the compound command beginning at _fn() and ending at SCRIPT.

  4. When _fn is called, sh must first interpret then map to a descriptor the file defined in <<SCRIPT...SCRIPT before invoking _fn as a special built-in utility because SCRIPT is _fn's <input.

  5. The strings output by printf and command are written out to _fn's standard-out >&1 - which is redirected to the current shell's ARGV0 - or $0.

  6. cat concatenates its <&0 standard-input file-descriptor - SCRIPT - over the > truncated current shell's ARGV0 argument, or $0.

  7. Completing its already read-in current compound command, sh execs the executable - and newly rewritten - $0 argument.

From the time ./file is called until its contained instructions specify that it should be execd again, sh reads it in a single compound command at a time as it executes them, while ./file itself does nothing at all except happily accept its new contents. The files that are actually at work are /usr/bin/sh, /usr/bin/cat, /tmp/sh-something-or-another.

THANKS, AFTER ALL

So when @jw013 specifies that:

A file that takes input and produces output is fine...

...amidst his erroneous criticism of this answer, he is actually unwittingly condoning the only method used here, which basically works out to just:

cat <new_file >old_file

ANSWER

All the answers here are good, but none of them are fully correct. Everyone seems to claim you cannot dynamically and permanently path your #!bang. Here's a demonstration of setting up a path independent shebang:

DEMO

{   cat >|./file
    chmod +x ./file
    ./file
} <<\FILE 
#!/usr/bin/sh
_rewrite_me() { printf '#!' ; command -v zsh
        ${out+cat} ; ${out+:} . /dev/fd/0 >&2
} <<\SCRIPT >|${out-/dev/null}
        printf "
        \$0    :\t$0
        lines :\t$((c=$(wc -l <$0)))
        !bang :\t$(sed 1q "$0")
        shell :\t"$(printf `ps -o args= -p $$`)\\n\\n
        sed -n "1,2{=;p};$((c-1)),\${=;p}" "$0" |
                sed -e 'N;s/\n/ >\t/' -e 4a\\...
SCRIPT
_rewrite_me ; out=$0 _rewrite_me ; exec $0
FILE

OUTPUT

        $0    : ./file
        lines : 13
        !bang : #!/usr/bin/sh
        shell : /usr/bin/sh

1 >     #!/usr/bin/sh
2 >     _rewrite_me() { printf '#!' ; command -v zsh
...
12 >    SCRIPT
13 >    _rewrite_me ; out=$0 _rewrite_me ; exec $0

        $0    : /home/mikeserv/file
        lines : 8
        !bang : #!/usr/bin/zsh
        shell : /usr/bin/zsh

1 >     #!/usr/bin/zsh
2 >             printf "
...
7 >             sed -n "1,2{=;p};$((c-1)),\${=;p}" "$0" |
8 >                     sed -e 'N;s/\n/ >\t/' -e 4a\\...

You see? We just make the script overwrite itself. And it only ever happens once after a git sync. From that point on it's got the right path in the #!bang line.

Now almost all of that up there is just fluff. To do this safely you need:

  1. A function defined at the top and called at the bottom that does the writing. This way we store everything we need in memory and ensure the entire file is read in before we begin writing over it.

  2. Some way of determining what the path should be. command -v is pretty good for that.

  3. Heredocs really help because they're actual files. They'll store your script in the meantime. You can use strings as well but...

  4. You have to make sure that the shell reads in the command that overwrites your script in the same command list as the one that execs it.

Look:

{   cat >|./file
    chmod +x ./file
    ./file
} <<\FILE 
#!/usr/bin/sh
_rewrite_me() { printf '#!' ; command -v zsh
        ${out+cat} ; ${out+:} . /dev/fd/0 >&2
} <<\SCRIPT >|${out-/dev/null}
        printf "
        \$0    :\t$0
        lines :\t$((c=$(wc -l <$0)))
        !bang :\t$(sed 1q "$0")
        shell :\t"$(printf `ps -o args= -p $$`)\\n\\n
        sed -n "1,2{=;p};$((c-1)),\${=;p}" "$0" |
                sed -e 'N;s/\n/ >\t/' -e 4a\\...
SCRIPT
_rewrite_me ; out=$0 _rewrite_me
exec $0
FILE

Notice that I only moved the exec command down one line. Now:

#OUTPUT
        $0    : ./file
        lines : 14
        !bang : #!/usr/bin/sh
        shell : /usr/bin/sh

1 >     #!/usr/bin/sh
2 >     _rewrite_me() { printf '#!' ; command -v zsh
...
13 >    _rewrite_me ; out=$0 _rewrite_me
14 >    exec $0

I don't get the second half of the output because the script can't read in the next command. Still, because the only command missing was the last:

cat ./file

#!/usr/bin/zsh
        printf "
        \$0    :\t$0
        lines :\t$((c=$(wc -l <$0)))
        !bang :\t$(sed 1q "$0")
        shell :\t"$(printf `ps -o args= -p $$`)\\n\\n
        sed -n "1,2{=;p};$((c-1)),\${=;p}" "$0" |
                sed -e 'N;s/\n/ >\t/' -e 4a\\...

The script came through as it should have - mostly because it was all in the heredoc - but if you don't plan it right you can truncate your filestream, which is what happened to me above.

8
  • The downvote is because self-modifying code is generally considered bad practice. Back in the old days of tiny assembly programs it was a clever way to reduce conditional branches and improve performance, but nowadays the security risks outweigh the advantages. Your approach would not work if the user who ran the script did not have write privileges on the script.
    – jw013
    Commented Apr 30, 2014 at 15:49
  • @jw013 Obviously my approach to install or update an executable script would not work if the person attempting to install or update the script did not have permissions to install or update the script. in fact, that is specifically what makes this answer better than every other answer here - it can provide an accurate #!bang line as necessary and only needs any special permissions to do so at the first invocation - during installation. And, again, i will not simply take your word for it that self modifying code is bad practice - please see man command for a contradictory opinion.
    – mikeserv
    Commented Apr 30, 2014 at 15:55
  • please see man command for a contradictory opinion - not finding one. Can you direct me to the specific section / paragraph you were talking about?
    – jw013
    Commented Apr 30, 2014 at 16:04
  • @jw013 - my mistake, it's in man sh - search for 'command -v'. I knew it was in one of the man pages i was looking at the other day.
    – mikeserv
    Commented Apr 30, 2014 at 16:11
  • I assume this is the command -v example you were talking about from man sh. That is a normal-looking installer script, and not a self-modifying one. Even self-contained installers only contain pre-modification input, and output their modifications somewhere else. They don't rewrite themselves the way you are recommending.
    – jw013
    Commented Apr 30, 2014 at 17:37
1

Here is one way to have a self modifying script that fix its shebang. This code should be prepended to your actual script.

#!/bin/sh
# unpatched

PATH=`PATH=/bin:/usr/bin:$PATH getconf PATH`
if [ "`awk 'NR==2 {print $2;exit;}' $0`" = unpatched ]; then
  [ -z "`PATH=\`getconf PATH\`:/usr/local/bin:/some/long/path/to/the/right:$PATH command -v zsh`" ] && { echo "zsh not found"; exit 1; }
  cp -- "$0" "$0.org" || exit 1
  mv -- "$0" "$0.old" || exit 1
  (
    echo "#!`PATH=\`getconf PATH\`:$PATH command -v zsh`" 
    sed -n '/^##/,$p' $0.old
  ) > $0 || exit
  chmod +x $0
  rm $0.old
  sync
  exit
fi
## Original script starts here

Some comments:

  • It should be run once by someone who has the rights to create and delete files in the directory where the script is located.

  • It only uses legacy bourne shell syntax as, despite popular belief, /bin/sh is not guaranteed to be a POSIX shell, even en POSIX compliant OSes.

  • It set the PATH to a POSIX compliant one followed by a list of possible zsh locations to avoid picking a "phony" zsh.

  • If for some reason, a self modifying script is unwelcomed, it would be trivial to distribute two scripts instead of one, the first one being the one you want to patch and the second one, the one I suggested slightly modified to process the former.

6
  • The /bin/sh point is a good one - but in that case do you need a premodified #! at all? And isn't awk just as likely to be phony as zsh is?
    – mikeserv
    Commented May 5, 2014 at 0:46
  • @mikeserv Answer updated to call the POSIX awk. The premodified shebang is there to prevent the script to be interpreted by a non bourne compatible shell should it be your login shell.
    – jlliagre
    Commented May 5, 2014 at 6:25
  • Makes sense. I upvoted it because it works, it sticks to the book and it demonstrates sound understanding of possible shell environment/file handling - especially the backup files you use, which is all GNU's sed -i does anyway. I personally think the $PATH problem noted in the comments on another answer and which you address as safely as i can figure here in a few lines is better handled by simply and explicitly defining dependencies and/or rigorous and explicit testing - for instance, now getconf might be phony, but the chances are near nil, same as they were for zsh and awk.
    – mikeserv
    Commented May 5, 2014 at 6:35
  • @mikeserv, script modified to reduce the risk to call a bogus getconf.
    – jlliagre
    Commented May 5, 2014 at 8:08
  • $(getconf PATH) is not Bourne. cp $0 $0.old is zsh syntax. The Bourne equivalent would be cp "$0" "$0.old" though you'd want cp -- "$0" "$0.old" Commented May 19, 2014 at 19:44

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .