[T]he behavior seems consistent between all POSIX-complaint shells. I don't see the need the need for wiggle room here.
You aren't looking deeply enough.
Back in the 1980s, this mechanism was not de facto standardized. Although Dennis Ritchie had implemented it, that implementation had not reached the public in the AT&T side of the universe. It was effectively only publicly available and known in BSD; with executable shell scripts not available on AT&T Unix. Thus it was not reasonable to standardize it. The state of affairs is exemplified by this contemporary doco, one of many such:
Note that BSD allows files which begin with — Stephen Frede (1988). "Programming on System X Release Y". Australian Unix Systems User Group Newsletter. Volume 9. Number 4. p. 111.
#! interpreter to be executed directly, while SysV allows only a.out files to be executed directly. This means that an instance of one of the
exec…() routines in a BSD program may have to be changed under SysV to execute the interpreter (typlically
/bin/sh) for that program instead.
An important point here is that you are looking at shells, whereas the existence of executable shell scripts is actually a matter for the
exec…() functions. What shells do includes the precursors of the executable script mechanism, still to be found in some shells even today (and also nowadays mandated for the
exec…p() subset of functions), and is somewhat misleading. What the standard needs to address in this regard is how
exec…() on an interpreted script works, and at the time that POSIX was originally created it simply did not work in the first place across a major part of the spectrum of target operating systems.
A subordinate question is why this has not been standardized since, especially as the magic number mechanism for script interpreters had reached the public in the AT&T side of the universe and had been documented for
exec…() in the System 5 Interface Definition, by the turn of the 1990s:
An interpreter file begins with a line of the form—
# ! pathname [arg]where pathname is the path of the interpreter, and arg is an optional argument.
exec an interpreter file, the system
execs the specified interpreter.
exec. System V Interface Definition. Volume 1. 1991.
Unfortunately, the behaviour remains today almost as widely divergent as it was in the 1980s and there is no truly common behaviour to standardize. Some Unices (famously HP-UX and FreeBSD, for examples) do not support scripts as interpreters for scripts. Whether the first line is one, two, or many elements separated by whitespace varies between MacOS (and versions of FreeBSD before 2005) and others. The maximum supported path length varies.
␀ and characters outwith the POSIX portable filename character set are tricky, as are leading and trailing whitespace. What the 0th, 1st, and 2nd argument end up being is also tricky, with significant variation across systems. Some currently POSIX-conformant but non-Unix systems still do not support any such mechanism, and mandating it would convert them into no longer being POSIX conformant.