Why are there so many "sh compatible" shells?
The Bourne shell was first publicly released in 1979 as part of Unix V7. Since pretty much every Unix and Unix-like system descends from V7 Unix — even if only spiritually — the Bourne shell has been with us "forever."1
The Bourne shell actually replaced an earlier shell, retronymed the Thompson shell, but it happened so early in Unix's history that it's all but forgotten today. The Bourne shell is a superset of the Thompson shell.2
Both the Bourne and Thompson shells were called
sh. The shell specified by POSIX is also called
sh. So, when someone says
sh-compatible, they are handwavingly referring to this series of shells. If they wanted to be specific, they'd say "POSIX shell" or "Bourne shell."3
The POSIX shell is based on the original version of the Korn shell, which in turn was meant to replace the Bourne shell on AT&T Unix, leapfrogging the BSD C shell in terms of features.4 To the extent that
ksh is the ancestor of the POSIX shell, most Unix and Unix-like systems include some form of
ksh today. (The exceptions are generally tiny embedded systems, which can't afford the space a complete POSIX shell takes.)
That said, the Korn shell — as a thing distinct from the POSIX shell — never really became popular outside the commercial Unix world. This is because its rise corresponded with the early years of Unix commercialization, so it got caught up in the Unix wars. BSD Unixes eschewed it in favor of the C shell, and its source code wasn't freely available for use in Linux when it got started.5 So, when the early Linux distributors went looking for a command shell to go with their Linux kernel, they usually chose GNU Bash, one of those
sh-compatibles you're talking about.6
That early association between Linux and Bash pretty much sealed the fate of many other shells, including
tcsh. There are die-hards still using those shells today, but they're very much in the minority.7
All this history explains why the creators of relative latecomers like
yash chose to make them
sh-compatible: Bourne/POSIX compatibility is the minimum a shell for Unix-like systems must provide in order to gain widespread adoption.
In many systems, there is a distinction between the interactive command shell and
/bin/sh, which is still the standard way to run shell scripts. The exact nature of
/bin/sh varies greatly between Unix and Unix-like systems. It may be:
The original Bourne shell. This is common in older UNIX® systems.8
The Almquist shell, an open source Bourne/POSIX shell clone with many forks, which vary in their degree of POSIX compliance.
This is a common choice in embedded Linuxes, since many of them are based on BusyBox, and BusyBox includes a fork of
Debian and Ubuntu famously shifted from Bash as
/bin/sh to an Almquist derivative called
Most BSDs have used an Almquist derivative as
/bin/sh since the early 1990s, including FreeBSD and NetBSD.
A POSIX-certified shell. This is common in SVR4 and newer UNIX® systems.
GNU Bash, which disables its non-POSIX extensions when called as
This choice is typical on desktop and server variants of Linux, except for Debian and its derivatives. Mac OS X also does this.
A classic trap on these systems is to write a shell script with a
#!/bin/sh shebang line at the top. Bash doesn't have its full power when run as
sh, but it is still more capable than
/bin/sh on many other systems. Such scripts often fail when moved to another OS.
A shell with
ksh93 POSIX extensions, as in OpenBSD. This is not common; you should not expect
ksh93 features in
I used "shell script" above as a generic term meaning Bourne/POSIX shell scripting. This is due to the ubiquity of Bourne family shells. To talk about scripting on other shells, you need to give a qualifier, like "C shell script." Even on systems where a C family shell is the default interactive shell, it is better to use the Bourne shell for scripting.
It is telling that when Wikipedia classifies Unix shells, they group them into Bourne shell compatible, C shell compatible, and "other."
This diagram may help:
(Click for SVG version, 29 kiB, or view full-size PNG version, 211 kiB.)
What would it mean to be "sh incompatible"?
Someone talking about an
sh-incompatible thing typically means one of three things:
They are referring to one of those "other" shells.9
They are making a distinction between the Bourne and C shell families.
They are talking about some specific feature in one Bourne family shell that isn't in all the other Bourne family shells.
zsh in particular have many features that don't exist in the older "standard" shells.
(Those three are also mutually-incompatible in a lot of ways, once you get beyond the shared POSIX/
Why are they different?
For the same reasons all "New & Improved!" things are different:
The improved version could only be improved by breaking backwards compatibility.
Someone thought of a different way for something to work, which they like better, but which isn't the same way the old one worked.
Someone tried reimplementing an old standard without completely understanding it, so they messed up and created an unintentional difference.
Footnotes and Asides:
Early versions of BSD Unix were just add-on software collections for V6 Unix. Since the Bourne shell wasn't added to AT&T Unix until V7, BSD didn't technically start out having the Bourne shell. BSD's answer to the primitive nature of the Thompson shell was the C shell.
Nevertheless, the first standalone versions of BSD (2.9BSD and 3BSD) were based on V7 or its portable successor UNIX/32V, so they did include the Bourne shell.
(The 2BSD line turned into a parallel fork of BSD for Digital's PDP minicomputers, while the 3BSD and 4BSD lines went on to take advantage of newer computer types like Vaxen and Unix workstations. 2.9BSD was essentially the PDP version of 4.1cBSD; they were contemporaneous, and shared code. PDPs didn't just disappear when the VAX arrived, so the 2BSD line is still shambling along.)
It is safe to say that the Bourne shell was everywhere in the Unix world by 1983. That's a good approximation to "forever" in the computing industry. MS-DOS got a hierarchical filesystem that year (awww, how cuuute!) and the first 24-bit Macintosh with its 9" B&W screen — not grayscale, literally black and white — wouldn't come out until early the next year.
The Thompson shell was quite primitive by today's standards. It was only an interactive command shell, rather than the script programming environment we expect today. It did have things like pipes and I/O redirection, which we think of as prototypically part of a "Unix shell," so that we think of the MS-DOS command shell as getting them from Unix.
The Bourne shell also replaced the PWB shell, which added important things to the Thompson shell like programmability (
while) and an early form of environment variables. The PWB shell is even less well-remembered than the Thompson shell since it wasn't part of every version of Unix.
When someone isn't specific about POSIX vs Bourne shell compatibility, there is a whole range of things they could mean.
At one extreme, they could be using the 1977 Bourne shell as their baseline. An "
sh-compatible script" in this sense would mean it is expected to run perfectly on the true Bourne shell or any of its successors and clones:
Someone at the other extreme assumes the POSIX shell as a baseline instead. We take so many POSIX shell features as "standard" these days that we often forget that they weren't actually present in the Bourne shell: job control, command history, aliases, command line editing, etc.
Although the Korn shell has roots going back to the early 1980s, AT&T didn't ship it in Unix until System V Release 4 in 1988. Since so many commercial Unixes are based on SVR4, this put
ksh in pretty much every relevant commercial Unix from the late 1980s onward.
(A few weird Unix flavors based on SVR3 and earlier held onto pieces of the market past the release of SVR4, but they were the first against the wall when the revolution came.)
1988 is also the year the first POSIX standard came out, with its Korn shell based "POSIX shell." Later, in 1993, an improved version of the Korn shell came out. Since POSIX effectively nailed the original in place,
ksh forked into two major versions:
ksh93, named after the years involved in their split.
ksh88 is not entirely POSIX-compatible, though the differences are small, so that some versions of the
ksh88 shell were patched to be POSIX-compatible. (This from an interesting interview on Slashdot with Dr. David G. Korn. Yes, the guy who wrote the shell.)
ksh93 is a fully-compatible superset of the POSIX shell, and continues to be enhanced. Its versions are denoted by suffixes, rather than by increasing year numbers. The current release is
ksh93u+, as I write this, with
ksh93v in beta test.
Systems that include a Korn shell as a separate thing from the POSIX shell usually make it available as
/bin/ksh, though sometimes it is hiding elsewhere.
When we talk about
ksh or the Korn shell by name, we are talking about
ksh93 features that distinguish it from its backwards-compatible Bourne and POSIX shell subsets. You rarely run across the pure
AT&T kept the Korn shell source code proprietary until March 2000. By that point, Linux's association with GNU Bash was very strong. Bash and
ksh93 each have advantages over the other, but at this point inertia keeps Linux tightly associated with Bash.
As to why the early Linux vendors most commonly choose GNU Bash over
pdksh, which was available at the time Linux was getting started, I'd guess it's because so much of the rest of the userland also came from the GNU project. Bash is also somewhat more advanced than
pdksh, since the Bash developers do not limit themselves to copying Korn shell features.
pdksh stopped about the time AT&T released the source code to the true Korn shell. There are two main forks that are still maintained, however: the OpenBSD
pdksh and the MirBSD Korn Shell,
I find it interesting that
mksh is the only Korn shell implementation currently packaged for Cygwin.
GNU Bash goes beyond POSIX in many ways, but you can ask it to run in a more pure POSIX mode.
tcsh was usually the default interactive shell on BSD Unixes through the early 1990s.
Being a BSD variant, early versions of Mac OS X were this way, through Mac OS X 10.2 "Jaguar". OS X switched the default shell from
tcsh to Bash in OS X 10.3 "Panther". This change did not affect systems upgraded from 10.2 or earlier. The existing users on those converted systems kept their
FreeBSD claims to still use
tcsh as the default shell, but on the FreeBSD 10 VM I have here, the default shell appears to be one of the POSIX-compatible Almquist shell variants. This is true on NetBSD as well.
OpenBSD uses a fork of
pdksh as the default shell instead.
The higher popularity of Linux and OS X makes some people wish FreeBSD would also switch to Bash, but they won't be doing so any time soon for philosophical reasons. It is easy to switch it, if this bothers you.
There are a couple of ways to run the original Bourne shell on your computer.
One is the heirloom Bourne shell project, which includes a portable version of the original Bourne shell for modern systems, released in 2005 through the now-defunct OpenSolaris project.
The other is the Computer History Simulation Project, which provides software and instructions for running the original Unix V7 binaries in the SIMH PDP-11 simulator, which runs on pretty much every modern computer, not just Unix-like ones. SIMH even runs on Android and on iOS.
Occasionally you encounter a
rc/es adherent, but they're even rarer than C shell fans.
rc family of shells isn't commonly used on Unix/Linux systems, but the family is historically important, which is how it earned a place in the diagram above.
rc is the standard shell of the Plan 9 from Bell Labs operating system, a kind of successor to 10th edition Unix, created as part of Bell Labs' continued research into operating system design. It is incompatible with both Bourne and C shell at a programming level; there's probably a lesson in there.
The most active variant of
rc appears to be the one maintained by Toby Goodwin, which is based on the Unix
rc clone by Byron Rakitzis.