Could someone explain this shebang line which uses sh and then does exec perl?

Question

This is from Programming Perl, Fourth Edition. It's about executing a Perl script.

Finally, if you are unfortunate enough to be on an ancient Unix system that doesn’t support the magic #! line, or if the path to your interpreter is longer than 32 characters (a built-in limit on many systems), you may be able to work around it like this:

#!/bin/sh -- # perl, to stop looping
eval 'exec /usr/bin/perl -S $0 ${1+"$@"}'
if 0;

Can you explain step by step what's going on here? I'm trying to make it work or encompass, but to no avail.

At execution of just the above, I'm getting this:

/bin/sh: 0: Illegal option --

Answer 1

The idea is that the eval command is valid in both shell and in Perl, with the difference that the newline terminates the command in shell, but not in Perl. Instead, Perl reads the following line, which adds the condition if 0, effectively negating the whole command.

(Perl supports a few of such "backwards" structures for shorthand, e.g. you can write next if $_ == 0 instead of if ($_ == 0) { next }, or print for @a instead of for (@a) { print }.)

If the script is started by a shell, the shell processes the eval, and replaces it with the Perl interpreter, giving it the script name ($0) and it's arguments ($@) as parameters.

Then Perl runs, reads the eval, skips it (because of the if 0), and then goes on to execute the rest of the script.

---

That's how it should work. In practice, you get the error because of two things: 1) Perl reads the hashbang line itself, and 2) the way Linux processes the hashbang lines.

When Perl runs a script with a hashbang, it doesn't really take it just as a comment. Instead, it interprets any options to Perl given in the hashbang (you can have #!/usr/bin/perl -Wln etc.) but it also checks the interpreter and executes it if the script isn't supposed to be run by Perl!

Try e.g. this:

$ cat > hello.sh
#!/bin/bash
echo $BASH_VERSION 
$ perl hello.sh
4.4.12(1)-release

That actually runs Bash.

So, the comment #perl is there to tell Perl that yes, this is actually supposed to be run by Perl, so that it doesn't start a shell again.

However, Linux gives everything after the interpreter name as a single argument, so when you run the script, it runs /bin/sh, with the two arguments -- # perl, to stop looping, and scriptname.pl. The first one starts with a dash, but isn't exactly --, so both Dash and Bash try to interpret it as options. It's not a valid one, so you get an error, similarly as if you tried to run bash ---, or bash "-- #perl".

On other systems that split the arguments in the hashbang line, -- #perl would give the shell --, #perl, and it would try to look for a file called #perl. This again wouldn't work. But apparently there are/have been some systems that take # signs as comment markers in the #! line, and/or only pass on the first argument. (See Sven Mascheck's page on the matter.) On those systems, it might work.

---

A somewhat better working one is given in perlrun (adapted):

#!/usr/bin/perl
eval 'exec /usr/bin/perl -S $0 ${1+"$@"}'
    if $running_under_some_shell;
print("Perl $^V\n");

Running that with bash ./script.pl actually runs Perl and prints (e.g.) Perl v5.24.1. ($running_under_some_shell is just an undefined variable, which defaults to falsy. if 0 would be cleaner but not as descriptive.)

---

Like it says on your quote, all of this is only required on ancient systems where #! doesn't work properly. In some, it's not supported at all, and asking a shell to run a non-binary always runs it in the shell. In others, there's a limit on the path length in the hashbang line, so #!/really/veeeery/long/path/to/perl wouldn't work.

Just put the #!/usr/bin/perl hashbang there on any modern system.

Answer 2

Solve /bin/sh: 0: Illegal option --

The simplest solution to avoid the error is to have only one argument to the sh call (on some systems) and also seems to be to add the option -x to the perl call^[d]. And I say seems because it depends on the system, kernel, shell and perl used to call the script.

So, better use something like:

#!/bin/sh --
eval 'exec perl -x -S $0 ${1+"$@"}'

#!/usr/bin/perl
print("Perl $^V\n");

In the script above the only perl executed command is to print the version.

Short Description

The only reason to use something like this is explained in the book (page 21):

Finally, if you are unfortunate enough to be on an ancient Unix system that doesn’t support the magic #! line, or if the path to your interpreter is longer than 32 characters (a built-in limit on many systems), you may be able to work around it ......

It is an ugly and convoluted way to replace the #!/usr/bin/perl call.

In the same page 21 is the script you posted. The next script about this issue, is in page 577 (keep reading)

Note: The inclusion of the perl name (word) in the first line is only needed if the script is at any time going to be called as perl ./script. In that case, that word will stop perl from looping. In fact, the second example from the book (page 577) is:

#!/bin/sh -- # -*- perl -*- -p
eval 'exec perl -S $0 ${1+"$@"}'
if 0;

In which the comment part of (# -- perl -- -p) is explained in the book as being interpreted by perl (and mostly ignored).

However, that is assuming that the shell call #!/bin/sh will allow more than one (--) argument. That is not true in some systems. It is better moved to the next line (if required for emacs) and/or removed:

#!/bin/sh -- # 
# -*- perl -*- -p
eval 'exec perl -S $0 ${1+"$@"}'
if 0;

A portable way to call perl

To understand the basics of it, the first concept to understand is that the script *may** be processed twice, once by a shell, then by the interpreter called by the shell, in this case, perl.

Understand that the script may be called as sh script, bash ./script, simply as ./script, or, if the path include the present working directory, as script and even as /usr/bin/perl script or as perl script. In DOS, Unix, Windows, Linux, etc.

How to make a call to perl that work in all systems?

Shell side

This code (cat ./script):

#!/bin/sh -- 
eval 'exec perl -S $0 ${1+"$@"}'
if 0;

Would be interpreted by almost any shell (either called as ./script or shell ./script as follows:

1. The kernel would read the first line, identify it as a shebang and start the shell executable that is placed in the /bin/sh file (or link). Also, the kernel will accept and process -- as an argument meaning "end of options"^[a][b]. That is the line:

   #!/bin/sh --
   

2. The shell (assuming there is a shell in /bin/sh) will read and execute the first (non-commented) line:

   eval 'exec perl -S $0 ${1+"$@"}'
   

   The first word (eval) will convert the list of arguments into a command line to execute removing one layer of quoting. That is, it will be converted to this command line:

   exec perl -S $0 ${1+"$@"}
   

   The first word of this (new) command line (that is: exec) will tell the kernel to replace the running process with the executable perl (searched in the PATH) and the arguments that result from expanding the rest of the command line.

The first argument (-S) will tell perl to also search for the executable script that is named in the next argument ($0). The $0 argument is (usually) the name of the previously loaded shell script (containing the code you posted aka ./script).

The next argument: ${1+"$@"} means: if arg $1 exists and is not null (${1+...}) replace this whole argument with the result of the expansion of "$@". The expansion of "$@" is the list of all the arguments to the loaded script.

You can see what arguments are given at this command line by testing:

    #!/bin/sh -
    eval 'exec /usr/bin/printf "<%s> " -S $0 ${1+"$@"}'

Calling this script:

    $  ./script one two t33 f44 "f55 s66"
    <-S> <./script> <one> <two> <t33> <f44> <f55 s66>

So, that line in the original script will call perl (equivalently) as:

    $ perl -S ./script one two t33 f44 "f55 s66"

That call to perl replaces the shell script being executed (exec).

Perl side

It is then perl that loads the script and re-interprets its content.
Comments in Perl also start with a #
Thus, the first line (shebang) is a comment to Perl.

The second line: eval 'exec perl -S $0 ${1+"$@"}' would be interpreted by perl (as eval is a valid Perl command) if the next line if 0; where not there. This third line makes Perl not execute the second line. A very long way of writing a nop.

The rest of the script is then interpreted and executed by Perl.

Note that all that construct is a very complex way of writing:

#!/usr/bin/perl

Why is it really needed?

To Set PATH

A "hello world" example similar to and using the -x perl option as usually recommended in the perldoc site.

Adding the -x call to perl will ensure that perl will itself search for a #!/usr/bin/perl shebang (which could be many lines below the first line) to start executing the script. This -x will also make the if 0 superfluous.

#!/bin/sh -
eval 'PATH="/usr/bin:/bin:/usr/local/bin:$PATH";: \
;exec perl -x -S -- "$0" ${1+"$@"};#'if 0;

#!/bin/perl -w
# Above is magic header ... real Perl code begins here
use strict;
use warnings;
print "hello world!\n";

That will set a quite more portable PATH to search for the perl executable. More variables could be set to make the call to perl more reliable (if needed).

^[d]The -x perl option will search for the shebang line with a perl name: #!/bin/perl -w to start interpreting the script from perl. That makes the separation between shell and perl quite more clear.

This script will work if it is called either as:

$ ./script
hello world!

$ sh ./script
hello world!

$ perl ./script
hello world!

but will fail if called from a csh shell. That is solved by adding eval '(exit $?0)' as a test and a new exec as this:

#!/bin/sh --
eval '(exit $?0)' && eval 'PATH="/usr/bin:/bin:/usr/local/bin:$PATH";: \
     ;exec perl -x -S -- "$0" ${1+"$@"};#'if 0;
exec 'exec perl -x -S -- "$0" $argv:q;#'.q

A Magic Header for Starting Perl Scripts
A related long description.

^[a]The whole string after the #!/bin/sh may be interpreted as only one argument. In such case, the whole string -- # perl, to stop looping could be reported as an error. As it is if bash directly loads the script when executing ./script (while the running shell is bash). Read more detail in search for "Splitting arguments".

^[b]In the very rare circumstances that it is useful to use a -- It is more portable to use a - instead as explained in this question.