3

I would like to put shebang #!/bin/sh -eufo pipefail in my script. But there're several things strange:

  1. The script would fail with that shebang in FreeBSD but not when run on MacOS
  2. on FreeBSd, the same shebang works when directly executed from command line (also /bin/sh).
>>> sh -eufo pipefail -c 'echo hi'  # this works
hi

>>> cat <<EOF > script                                                                                      
#!/bin/sh -eufo pipefail
echo hi
EOF

>>> chmod +x ./script 
>>> ./script  # this doesn't work on FreeBSD but works on MacOS
Illegal option -o ./script

>>> cat ./script 
#!/bin/sh -eufo pipefail
echo hi

>>> uname -a
FreeBS 11.3-RELEASE-p7
6
  • You forgot to try sh '-eufo pipefail' ./script. – JdeBP Aug 22 '20 at 11:41
  • My impression is that you do not get a 100% usable answer because your question does not give the needed information. ash e.g. does not support set -o pipefail and people cannot help you because you hide the error messages you got. – schily Aug 22 '20 at 22:32
  • I did not hide anything. I spent time coming up with a easy to reproduce code snippet and included the output in the question in full fidelity. I thought that should be pretty obvious. – KFL Aug 22 '20 at 22:37
  • Oh sorry, I missed the error message illeal option. So it it obvious that the problem is not the #! but rather the fact that you are using a non-standard option that is not supported by ash on FreeBSD. – schily Aug 23 '20 at 6:32
  • 1
    @schily, note that it says Illegal option -o ./script, not e.g. Illegal option -o pipefail, which is what I get for e.g. Debian's dash when started with dash -o pipefail (command name + two args). And I know, dash isn't the FreeBSD sh, but an ash derivative nonetheless, so it might be related. – ilkkachu Aug 23 '20 at 9:25
1

MacOS still retains the old FreeBSD behaviour from before 2005. In 2005, there was a major change in the way that the FreeBSD kernel handled #! at the start of an executable file passed to execve(), to bring it more into line with some other operating system kernels, including Linux and the NetBSD kernel.

Commentary in the NetBSD kernel source code tries to paint this as a universal:

 * collect the shell argument.  everything after the shell name
 * is passed as ONE argument; that's the correct (historical)
 * behaviour.
kern/exec-script.c. NetBSD. lines 189 et seq..

It actually is not. Sven Mascheck did some testing about a decade ago and there are four basic behaviours, the AT&T Unix System 5 one having as much claim to being "correct historical" behaviour as the 4.2BSD one has:

  • Ignore the characters (before 4.2BSD and AT&T Unix System 5).
  • Pass the whole string in a single argument (4.2BSD, NetBSD, Linux and FreeBSD from 2005 onwards).
  • Split the string up by whitespace and pass it as multiple arguments (FreeBSD before 2005 and MacOS).
  • Split the string up by whitespace and pass just the first argument (AT&T Unix System 5 and Solaris)

I've only included the operating systems relevant to this answer in parentheses. M. Mascheck checked a lot more, as did Ahmon Dancy in discussion of FreeBSD Problem Report 16393. See the further reading for the full lists.

What brought things to a head in FreeBSD in 2005 was that, ironically, FreeBSD wasn't quite as simple as that. It had had a change introduced that was intended to make things written in popular books about Perl actually work: arguments were skipped after a comment character. The books had recommended things like:

#!/bin/sh -- # -*- perl -*-
— Larry Wall, Tom Christiansen, Jon Orwant (2000). Programming Perl: 3rd Edition. O'Reilly Media. ISBN 9780596000271. p. 488.

PR 16393 in 2000 was a way of making the kernel handle executable Perl scripts, written in the way that Larry Wall no less had said would work. However, it broke other stuff and didn't completely work.

There was some back and forth on this. Finally, in 2005 the mechanism to make Larry Wall et al.'s idea work was moved out of the kernel, which was made to behave compatibly with Linux, NetBSD, and 4.2BSD (rather than Solaris and AT&T Unix System 5) and made the responsibility of sh.

The behaviour since 2005 has thus been that the shell gets three arguments, the second argument being the entire tail of the #! line, and invoking your script directly with execve() is effectively the same as invoking:

sh '-eufo pipefail' ./script

It should be fairly obvious why the Almquist shell (which is what sh is on FreeBSD) is thinking that ./script is the option argument for the -o option, and that it is treating the pipefail part as further single-letter options collected behind - (which it hasn't got around to processing yet).

An also obvious alternative is to have set -o pipefail as the first command in the script, as pointed out at https://unix.stackexchange.com/a/533418/5132 for the Bourne Again shell. This was only added to the FreeBSD Almquist shell in 2019, however and thus is only available in very recent versions of FreeBSD. (The Debian Almquist shell has not yet had it added, as of 2020.)

Further reading

12
  • The table from Sven Mascheck is not correct... SunOS-4 of course supports suid scripts and introduced to let the kernel typicaly use /dev/fd/5 as argument instead of the shellscipt. /dev/fd/5 was opened by the kernel for security reasons. You however need to specify the shell option -p in the first line of the script as the shell forbids scripts where effective and real user ID differ. So for correct usability, you only ever need one single option from the #! line: -p. – schily Aug 22 '20 at 13:49
  • The question is about FreeBSD and MacOS, M. Schilling. – JdeBP Aug 22 '20 at 13:57
  • If people like to write something that works everywhere, it is a good practice to follow the common demnominator Mr. anonymous. And BTW: if you understand the historical background it is possible to understand an alleged limitation. – schily Aug 22 '20 at 14:05
  • 1
    hmm, what are the characters in "Ignore the characters" in the first point? Did those OS's ignore the whole hashbang line, ignore the characters after the first space in the line, or ignore characters after the second space,or ...? – ilkkachu Aug 22 '20 at 16:33
  • 1
    "It should be fairly obvious why the Almquist shell is thinking that ./script is the option argument for the -o option" -- actually, I don't find that obvious at all... Ksh and Zsh seem to interpret that as the -o option with an argument ␣pipefail (with a leading space), and as far as I can read getopts and getopt(), that's how they're supposed to interpret it. – ilkkachu Aug 22 '20 at 16:36
0

Even though your #! usage is non-portable because it uses more than the simple:

#!/bin/sh

or the commonly supported single argument as in:

#!/bin/sh -oneflag

The real problem is that you are using a non-portable option -o pipefail.

This is an option from ksh93 and bash that is not supported by other shells.

Background:

  • On MacOS,/bin/sh is bash that has been compiled in a specific way (e.g. to make escape sequences in echo work by default) to make it POSIX compliant. Since bash supports pipefail (see above), this works on MacOS.

  • On FreeBSD, /bin/sh is ash that does not support pipefail.

You have two possible ways to go:

  • Do not use set -o pipefail

  • Wait for two years and try again. This will most likely work, because we decided to add this option to the next POSIX standard (Issue-8) 10 months ago, see https://www.austingroupbugs.net/view.php?id=789 and since the next POSIX standard will be published in approx. one year, there is a big chance that FreeBSD will add support for set -o pipefail to ash soon.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.