Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I've seen the phrase "sh compatible" used usually in reference to shells. I'm not sure if it also applies to the programs that might be run from within shells.

What does it mean for a shell or other program to be "sh compatible"? What would it mean to be "sh incompatible"?

Edit: This question asking the difference between bash and sh is very relevant: http://stackoverflow.com/questions/5725296/difference-between-sh-and-bash

I'd still like a direct answer to what it means to be "sh compatible". A reasonable expectation might be that "sh compatible" means "implements the Shell Command Language" but then why are there so many "sh compatible" shells and why are they different?

share|improve this question
    
For a shell script, it refers to whether or not that script uses syntax compatible (runable) with the Bourne Shell (sh). For a different shell, it refers to whether or not that shell can run Bourne shell scripts. –  HalosGhost Jul 20 at 0:16

3 Answers 3

up vote 25 down vote accepted

Why are there so many "sh compatible" shells?

The Bourne shell has been with us since 1977, released as part of Unix V7. Since pretty much every Unix and Unix-like system descends from V7 Unix — even if only spiritually — the Bourne shell has been with us "forever."1

The Bourne shell actually replaced an earlier shell, retronymed the Thompson shell, but it happened so early in Unix's history that it's all but forgotten today. The Bourne shell is a superset of the Thompson shell.2

Both the Bourne and Thompson shells were called sh. The POSIX shell is also called sh. So, when someone says sh-compatible, they are handwavingly referring to this series of shells. If they wanted to be specific, they'd say "POSIX shell" or "Bourne shell."3

The POSIX shell is based on the original version of the Korn shell, which in turn was meant to replace the Bourne shell on AT&T Unix, leapfrogging the BSD C shell in terms of features.4 To the extent that ksh is the ancestor of the POSIX shell, most Unix and Unix-like systems include some form of ksh today. (The exceptions are generally tiny embedded systems, which can't afford the space a complete POSIX shell takes.)

That said, the Korn shell — as a thing distinct from the POSIX shell — never really became popular outside the commercial Unix world. This is because its rise corresponded with the early years of Unix commercialization, so it got caught up in the Unix wars. BSD Unixes eschewed it in favor of the C shell, and its source code wasn't freely available for use in Linux when it got started.5 So, when the early Linux distributors went looking for a command shell to go with their Linux kernel, they usually chose GNU Bash, one of those sh-compatibles you're talking about.6

That early association between Linux and Bash pretty much sealed the fate of many other shells, including ksh, csh and tcsh. There are die-hards still using those shells today, but they're very much in the minority.7

All this history is why a relative latecomer like zsh ends up being sh-compatible. Bourne/POSIX compatibility is the minimum a shell for Unix-like systems must provide in order to gain widespread adoption.

In many systems, there is a distinction between the interactive command shell and /bin/sh, which is still the standard way to run shell scripts. The exact nature of /bin/sh varies greatly between Unix and Unix-like systems. It may be:

  • The original Bourne shell. This is common in older UNIX® systems.8

  • The Almquist shell, an open source Bourne/POSIX shell clone with many forks, which vary in their degree of POSIX compliance.

    This is a common choice in embedded Linuxes, since many of them are based on BusyBox, and BusyBox includes a fork of ash.

    Debian and Ubuntu famously shifted from Bash as /bin/sh to an Almquist derivative called dash.

    Most BSDs have used an Almquist derivative as /bin/sh since the early 1990s, including FreeBSD and NetBSD.

  • A POSIX-certified shell. This is common in SVR4 and newer UNIX® systems.

  • GNU Bash, which disables its non-POSIX extensions when called as sh.

    This choice is typical on desktop and server variants of Linux, except for Debian and its derivatives. Mac OS X also does this.

    A classic trap on these systems is to write a shell script with a #!/bin/sh shebang line at the top. Bash doesn't have its full power when run as sh, but it is still more capable than /bin/sh on many other systems. Such scripts often fail when moved to another OS.

  • A shell with ksh93 POSIX extensions, as in OpenBSD. This is not common; you should not expect ksh93 features in /bin/sh.

I used "shell script" above as a generic term meaning Bourne/POSIX shell scripting. This is due to the ubiquity of Bourne family shells. To talk about scripting on other shells, you need to give a qualifier, like "C shell script." Even on systems where a C family shell is the default interactive shell, it is better to use the Bourne shell for scripting.

It is telling that when Wikipedia classifies Unix shells, they group them into Bourne shell compatible, C shell compatible, and "other."

This diagram may help:

The Unix Shells, Bourne and C Shell Families

(Click for SVG version, 25 kiB, or view full-size PNG version, 173 kiB.)

What would it mean to be "sh incompatible"?

Someone talking about an sh-incompatible thing typically means one of three things:

  1. They are referring to one of those "other" shells.9

  2. They are making a distinction between the Bourne and C shell families.

  3. They are talking about some specific feature in one Bourne family shell that isn't in all the other Bourne family shells. ksh93, bash, and zsh in particular have many features that don't exist in the older "standard" shells.

    (Those three are also mutually-incompatible in a lot of ways, once you get beyond the shared POSIX/ksh88 base.)

Why are they different?

For the same reasons all "New & Improved!" things are different:

  • The improved version could only be improved by breaking backwards compatibility.

  • Someone thought of a different way for something to work, which they like better, but which isn't the same way the old one worked.

  • Someone tried reimplementing an old standard without completely understanding it, so they messed up and created an unintentional difference.


Footnotes and Asides:

  1. Early versions of BSD Unix were just add-on software collections for V6 Unix. Since the Bourne shell wasn't added to AT&T Unix until V7, BSD didn't technically start out having the Bourne shell. BSD's answer to the primitive nature of the Thompson shell was the C shell.

    Nevertheless, the first standalone versions of BSD (2.9BSD and 3BSD) were based on V7 or its portable successor UNIX/32V, so they did include the Bourne shell.

    (The 2BSD line turned into a parallel fork of BSD for Digital's PDP minicomputers, while the 3BSD and 4BSD lines went on to take advantage of newer computer types like Vaxen and Unix workstations. 2.9BSD was essentially the PDP version of 4.1cBSD; they were contemporaneous, and shared code. PDPs didn't just disappear when the VAX arrived, so the 2BSD line is still shambling along.)

    It is safe to say that the Bourne shell was everywhere in the Unix world by 1983. That's a good approximation to "forever" in the computing industry. MS-DOS got a hierarchical filesystem that year (awww, how cuuute!) and the first 24-bit Macintosh with its 9" B&W screen — not grayscale, literally black and white — wouldn't come out until early the next year.

  2. The Thompson shell was quite primitive by today's standards. It was only an interactive command shell, rather than the script programming environment we expect today. It did have things like pipes and I/O redirection, which we think of as prototypically part of a "Unix shell," so that we think of the MS-DOS command shell as getting them from Unix.

    The Bourne shell also replaced the PWB shell, which added important things to the Thompson shell like programmability (if, switch and while) and an early form of environment variables. The PWB shell is even less well-remembered than the Thompson shell since it wasn't part of every version of Unix and was replaced by the Bourne shell after just two years.

  3. When someone isn't specific about POSIX vs Bourne shell compatibility, there is a whole range of things they could mean.

    At one extreme, they could be using the 1977 Bourne shell as their baseline. An "sh-compatible script" in this sense would mean it is expected to run perfectly on the true Bourne shell or any of its successors and clones: ash, bash, ksh, zsh, etc.

    Someone at the other extreme assumes the POSIX shell as a baseline instead. We take so many POSIX shell features as "standard" these days that we often forget that they weren't actually present in the Bourne shell: job control, command history, aliases, command line editing, etc.

  4. Although the Korn shell has roots going back to the early 1980s, AT&T didn't ship it in Unix until System V Release 4 in 1988. Since so many commercial Unixes are based on SVR4, this put ksh in pretty much every relevant commercial Unix from the late 1980s onward.

    (A few weird Unix flavors based on SVR3 and earlier held onto pieces of the market past the release of SVR4, but they were the first against the wall when the revolution came.)

    1988 is also the year the first POSIX standard came out, with its Korn shell based "POSIX shell." Later, in 1993, an improved version of the Korn shell came out. Since POSIX effectively nailed the original in place, ksh forked into two major versions: ksh88 and ksh93, named after the years involved in their split.

    ksh88 is not entirely POSIX-compatible, though the differences are small, so that some versions of the ksh88 shell were patched to be POSIX-compatible. (This from an interesting interview on Slashdot with Dr. David G. Korn. Yes, the guy who wrote the shell.)

    ksh93 is a fully-compatible superset of the POSIX shell, and continues to be enhanced. Its versions are denoted by suffixes, rather than by increasing year numbers. The current release is ksh93v, as I write this.

    Systems that include a Korn shell as a separate thing from the POSIX shell usually make it available as /bin/ksh, though sometimes it is hiding elsewhere.

    When we talk about ksh or the Korn shell by name, we are talking about ksh93 features that distinguish it from its backwards-compatible Bourne and POSIX shell subsets. You rarely run across the pure ksh88 today.

  5. AT&T kept the Korn shell source code proprietary until March 2000. By that point, Linux's association with GNU Bash was very strong. Bash and ksh93 each have advantages over the other, but at this point inertia keeps Linux tightly associated with Bash.

    As to why the early Linux vendors most commonly choose GNU Bash over pdksh, which was available at the time Linux was getting started, I'd guess it's because so much of the rest of the userland also came from the GNU project. Bash is also somewhat more advanced than pdksh, since the Bash developers do not limit themselves to copying Korn shell features.

    Work on pdksh stopped about the time AT&T released the source code to the true Korn shell. There are two main forks that are still maintained, however: the OpenBSD pdksh and the MirBSD Korn Shell, mksh.

  6. GNU Bash goes beyond POSIX in many ways, but you can ask it to run in a more pure POSIX mode.

  7. csh/tcsh was usually the default shell on BSD Unixes through the early 1990s.

    Being a BSD variant, early versions of Mac OS X were this way, through Mac OS X 10.2 "Jaguar". OS X switched the default shell from tcsh to Bash in OS X 10.3 "Panther". This change did not affect systems upgraded from 10.2 or earlier. The existing users on those converted systems kept their tcsh shell.

    FreeBSD claims to still use tcsh as the default shell, but on the FreeBSD 10 VM I have here, the default shell appears to be one of the POSIX-compatible Almquist shell variants. This is true on NetBSD as well.

    OpenBSD uses a fork of pdksh as the default shell instead.

    The higher popularity of Linux and OS X makes some people wish FreeBSD would also switch to Bash, but they won't be doing so any time soon for philosophical reasons. It is easy to switch it, if this bothers you.

  8. There are a couple of ways to run the original Bourne shell on your computer.

    One is the heirloom Bourne shell project, which includes a portable version of the original Bourne shell for modern systems, released in 2005 through the now-defunct OpenSolaris project.

    The other is the Computer History Simulation Project, which provides software and instructions for running the original Unix V7 binaries in the SIMH PDP-11 simulator, which runs on pretty much every modern computer, not just Unix-like ones. SIMH even runs on Android and on iOS.

  9. Occasionally you encounter a fish or scsh adherent, but they're even rarer than C shell fans.

share|improve this answer

"sh compatible" refers to POSIX sh, the basic shell that is required to exist on all compatible systems. A sh-compatible script should work on any POSIX-compatible machine.

The reason it's necessary to say so is that commonly /bin/sh is a symlink to /bin/bash, which has let some Bashisms slip into scripts that declare themselves to use sh with #!/bin/sh. These scripts fail to work on systems that don't use bash as /bin/sh, including some commercial Unices forever, and Debian and derivatives recently.

In particular there's been a trend to use dash, the Debian Almquish Shell, as the default sh lately, because it's smaller and meant to be faster. That trend has highlighted a lot of those Bashisms that were in supposed sh scripts. Describing something as "sh compatible" indicates that it's explicitly intended to work with these systems by staying entirely within the POSIX-specified language — all shells will implement a superset of that functionality, so it's guaranteed to work everywhere, but their extensions aren't compatible with one another.

Different shells have their own development histories and diverged in different directions over time, as they added functions to help interactive use for their users, or scripting extensions like associative arrays. A "sh-incompatible" script would use some of these non-standard extension features, like Bash's [[ conditionals.

The non-POSIX features in bash and tcsh and zsh and all of the other current shells are useful, and there are plenty of occasions where you might want or need them. They just shouldn't be used in a script that declares itself to work with /bin/sh, because you can't rely on those features being in the base sh implementation on the system you're running on.

A script that does need to use, say, associative arrays, should ensure it's run with bash rather than sh:

#!/bin/bash
declare -A array

That will work anywhere with bash. Scripts that don't need the extended functionality and are meant to be portable should declare that they use sh and stick to the base shell command language.

share|improve this answer

For a little fun and edification, here is a shell that is sh-compatible:

cat /*dev/null; echo "Happy New Year"\!
cat <<c*/ /*dev/null | cat > /dev/null
c */ () {} /*
c */ main() { cat(); printf("Happy New Year!\n"); } /*
17      format('Happy New Year!')
        write (6,17)
        stop
        end
c*/

You can run it with sh, but also with csh, C, and fortran.

Now, of course, most shell scripts are not that esoteric, and are just compatible with sh, and perhaps with ksh, bash, etc.

Here is a makefile to try it out easily:

#FORTRAN=f77
FORTRAN=gfortran

all: happy which

happy:happy-c happy-f happy-sh happy-csh
    @  printf '(C)            '
    -@ ./happy-c || true
    @  printf '(FORTRAN)      '
    -@ ./happy-f
    @  printf '(Bourne Shell) '
    -@ sh ./happy-sh
    @  printf '(C Shell)      '
    -@ csh ./happy-csh

happy-c:happy.source Makefile
    @ cp happy.source happy-c.c
    @ $(CC) --no-warning -o happy-c happy-c.c

happy-f:happy.source Makefile
    @ cp happy.source happy-f.f
    @ $(FORTRAN) -o happy-f  happy-f.f 2> /dev/null

happy-sh:happy.source Makefile
    @ cp happy.source happy-sh
    @ chmod 755 happy-sh

happy-csh:happy.source Makefile
    @ cp happy.source happy-csh
    @ chmod 755 happy-csh

clean:
    -rm -f *.o *~ happy-csh happy-sh happy-f happy-f.f happy-c happy-c.c

which:
    @printf "\n"
    -bash   ./which_interpreter
    @printf "\n"
    -expect ./which_interpreter
    @printf "\n"
    -perl   ./which_interpreter
    @printf "\n"
    -python ./which_interpreter
    @printf "\n"
    -ruby   ./which_interpreter
    @printf "\n"
    -sh     ./which_interpreter
    @printf "\n"
    -tcsh   ./which_interpreter
    @printf "\n"
    -wish   ./which_interpreter
    @printf "\n"
    -TERM=dumb wine cmd.exe  ./which_interpreter < /dev/null
    @printf "\n"
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.