FreeBSD shebang error

Question

I would like to put shebang #!/bin/sh -eufo pipefail in my script. But there're several things strange:

1. The script would fail with that shebang in FreeBSD but not when run on MacOS
2. on FreeBSd, the same shebang works when directly executed from command line (also /bin/sh).

>>> sh -eufo pipefail -c 'echo hi'  # this works
hi

>>> cat <<EOF > script                                                                                      
#!/bin/sh -eufo pipefail
echo hi
EOF

>>> chmod +x ./script 
>>> ./script  # this doesn't work on FreeBSD but works on MacOS
Illegal option -o ./script

>>> cat ./script 
#!/bin/sh -eufo pipefail
echo hi

>>> uname -a
FreeBS 11.3-RELEASE-p7

Answer 1

MacOS still retains the old FreeBSD behaviour from before 2005. In 2005, there was a major change in the way that the FreeBSD kernel handled #! at the start of an executable file passed to execve(), to bring it more into line with some other operating system kernels, including Linux and the NetBSD kernel.

Commentary in the NetBSD kernel source code tries to paint this as a universal:

 * collect the shell argument.  everything after the shell name
 * is passed as ONE argument; that's the correct (historical)
 * behaviour.

^{— kern/exec-script.c. NetBSD. lines 189 et seq..}

It actually is not. Sven Mascheck did some testing about a decade ago and there are four basic behaviours, the AT&T Unix System 5 one having as much claim to being "correct historical" behaviour as the 4.2BSD one has:

• Ignore the characters (before 4.2BSD and AT&T Unix System 5).
• Pass the whole string in a single argument (4.2BSD, NetBSD, Linux and FreeBSD from 2005 onwards).
• Split the string up by whitespace and pass it as multiple arguments (FreeBSD before 2005 and MacOS).
• Split the string up by whitespace and pass just the first argument (AT&T Unix System 5 and Solaris)

I've only included the operating systems relevant to this answer in parentheses. M. Mascheck checked a lot more, as did Ahmon Dancy in discussion of FreeBSD Problem Report 16393. See the further reading for the full lists.

What brought things to a head in FreeBSD in 2005 was that, ironically, FreeBSD wasn't quite as simple as that. It had had a change introduced that was intended to make things written in popular books about Perl actually work: arguments were skipped after a comment character. The books had recommended things like:

#!/bin/sh -- # -*- perl -*-

^{— Larry Wall, Tom Christiansen, Jon Orwant (2000). Programming Perl: 3rd Edition. O'Reilly Media. ISBN 9780596000271. p. 488.}

PR 16393 in 2000 was a way of making the kernel handle executable Perl scripts, written in the way that Larry Wall no less had said would work. However, it broke other stuff and didn't completely work.

There was some back and forth on this. Finally, in 2005 the mechanism to make Larry Wall et al.'s idea work was moved out of the kernel, which was made to behave compatibly with Linux, NetBSD, and 4.2BSD (rather than Solaris and AT&T Unix System 5) and made the responsibility of sh.

The behaviour since 2005 has thus been that the shell gets three arguments, the second argument being the entire tail of the #! line, and invoking your script directly with execve() is effectively the same as invoking:

sh '-eufo pipefail' ./script

It should be fairly obvious why the Almquist shell (which is what sh is on FreeBSD) is thinking that ./script is the option argument for the -o option, and that it is treating the pipefail part as further single-letter options collected behind - (which it hasn't got around to processing yet).

An also obvious alternative is to have set -o pipefail as the first command in the script, as pointed out at https://unix.stackexchange.com/a/533418/5132 for the Bourne Again shell. This was only added to the FreeBSD Almquist shell in 2019, however and thus is only available in very recent versions of FreeBSD. (The Debian Almquist shell has not yet had it added, as of 2020.)

Further reading

• Pointers to some of the older history: https://unix.stackexchange.com/a/489688/5132
• Sven Mascheck. "Test results from various systems". The #! magic, details about the shebang/hash-bang mechanism on various Unix flavours. www.in-ulm.de/~mascheck.
• Garance A Drosihn (2005-02-23). Bug in #! processing — One More Time. freebsd-arch mailing list.
• ryand (2000-01-27). /bin/sh doesn't strip comments on shebang line. FreeBSD Problem Report 16393.
• Garance A Drosihn (2005). Updating note for May 28, 2005: Change in handling of shell-script options. people.freebsd.org/~gad.
• sh. BSD General Commands Manual. 2019-02-24. freebsd.org.
• Wolfram Schneider (2017-12-12). Get exit status of process that's piped to another: set -o pipefail is missing for /bin/sh. FreeBSD Problem Report 224270.
• Ibrahim Ghazal (2020-06-30). Status of set -o pipefail. Debian Almquist shell mailing list.

Answer 2

Even though your #! usage is non-portable because it uses more than the simple:

#!/bin/sh

or the commonly supported single argument as in:

#!/bin/sh -oneflag

The real problem is that you are using a non-portable option -o pipefail.

This is an option from ksh93 and bash that is not supported by other shells.

Background:

• On MacOS,/bin/sh is bash that has been compiled in a specific way (e.g. to make escape sequences in echo work by default) to make it POSIX compliant. Since bash supports pipefail (see above), this works on MacOS.

• On FreeBSD, /bin/sh is ash that does not support pipefail.

You have two possible ways to go:

• Do not use set -o pipefail

• Wait for two years and try again. This will most likely work, because we decided to add this option to the next POSIX standard (Issue-8) 10 months ago, see https://www.austingroupbugs.net/view.php?id=789 and since the next POSIX standard will be published in approx. one year, there is a big chance that FreeBSD will add support for set -o pipefail to ash soon.