1

I have been enthusiastic to shell scripting for some months, and found that the codes I write (in bash) do not work in some machines. This is very disappointing.

I realize that I should have learned the most portable shell scripting instead. This would, of course, be a bit painful in the beginning, but I believe it will pay off eventually. I hope my shell scripts will work almost everywhere, including the Linux family and the BSD family. (But I don't care if they run on Windows.)

Questions

  1. What's the most portable shell? EDIT: Please omit this question

  2. What are some serious disadvantages to use the most portable shell? EDIT: Please omit this question

  3. What else should I pay attention to if I want to write extremely portable codes? e.g. what core-utils and which versions should I use and avoid?

16
  • Your question is a bit missleading as it leaves open whether you like to know which shell runs on the largest number of platforms ot whether you are interested in a shell that does not implement POSIX extensions. – schily Dec 1 '19 at 23:00
  • 1
    I think the question is phrased slightly wrong; AIUI the question isn't about portability of different shells, but about writing shell scripts that are portable between different shells and shell environments. – Gordon Davisson Dec 2 '19 at 0:24
  • @GordonDavisson I did mean shells, but yeah also basic-utils. I will include that in the title. Thank you! – Student Dec 2 '19 at 1:34
  • 1
    I think you're still misunderstanding my point. Looking for a portable shell means you're looking for a shell (and utilities) you can bring with you and install on whatever system you happen to be using today; but AUIU you're actually interested in writing scripts that'll work on whatever shell (and utilities) happen to already be available on the system. That's not a matter of looking for a portable shell, but of looking for portable scripting practices. – Gordon Davisson Dec 2 '19 at 2:46
  • 2
    Shell Check is nice utility and website that can help write portable shell scripts. shellcheck.net – Christopher Dec 2 '19 at 12:46
7

Your bash scripts should work on any machine that has the bash shell installed, unless you use bash features that are too new for the version of the shell on the other system, like for example, trying to use name references (declare -n), associative arrays (declare -A), or the %(...)T format of printf (or any of a number of other different things) with the default bash on macOS (which is ancient; just install a newer bash with Homebrew in that case). Your scripts may also fail if they use external tools not available on some other system, or features of tools that may not be implemented, or that are implemented differently.

The most portable shell language on POSIX systems is the sh shell. This is described here: https://pubs.opengroup.org/onlinepubs/9699919799/idx/shell.html

The bash shell (and others) implement sh, with extensions. Some shells, like fish or zsh, are not even trying to be sh shells at all (but zsh provides a degree of "sh-emulation" via --emulate sh), while shells like dash does try to be more pure to the POSIX sh standard (but still with a few extensions).

In general, shell aspects that relate to syntax and built in features should be "portable" as long as the script is always executed by the same shell (and version of the shell, to some degree), even if that shell happens to be the fish shell. This is the same type of "portability" that you have with Python, Perl and Ruby scripts. There should however always be an sh shell available (usually installed as /bin/sh), which is usually bash, dash, or ksh running in some form of "compatibility mode".

The most portable utilities on a POSIX system are the POSIX utilities. These are described here: https://pubs.opengroup.org/onlinepubs/9699919799/idx/utilities.html

GNU utilities, the ones in coreutils, implement the POSIX utilities (which, as user schily points out in comments, does not mean that they are POSIX-compliant), and then expands upon them with features that are mainly for convenience. This is also true for the corresponding utilities on other Unix systems. Also note that the GNU coreutils package on Linux does not contain all utilities that POSIX specifies and that utilities such as find, sed, awk, and others, which are all POSIX utilities, are packaged separately.

As long as you stay with utilities that are POSIX and use their POSIX behaviour (in terms of options etc.), in scripts that use POSIX sh syntax, you are likely to be "mostly portable" across most Unix systems. But also note that there are non-POSIX utilities and non-POSIX extension to POSIX utilities that are still fairly portable, because they are commonly implemented. Examples of common non-POSIX utilities are pkill, tar, and gzip. Examples of non-POSIX extensions to POSIX utilities that are common to find is the -iname predicate of find and the fact that sed often can be made to understand extended regular expressions using -E.

You'll learn what extensions in implementations of what tools, running on what Unices, behaves in what way, by testing by yourself (in virtual machines for example), and by reading questions and answers on this site. For example, How can I achieve portability with sed -i (in-place editing)?

Also note that there are instances when the POSIX standard leaves a behaviour "unspecified", which mean that the same utility on two different Unix systems, using a POSIX option, may behave differently. An example of this is found here in Why permission denied upon symbolic link update to new target with permissions OK?

The disadvantage of trying to write pure POSIX shell code is that you miss out on some really useful features. I once tried to write the POSIX equivalent tests for find that GNU's -readable predicate tests for, which wasn't easy, and trying to signal a process by name using ps and grep instead of pkill is a topic of a few too many questions on this site. Likewise, trying to "stay POSIX", you would need to stay away from really useful things like perl, and you would not be able to do much of any administrative task portably (adding users, manage backups, transfer files between systems, etc.)

In general I prefer a pragmatic approach to getting things done, while understanding that my code may have to be modified to support other systems later (if it's ever used on other systems), rather than a purist approach. This does not stop me from writing scripts that I know will have as few portability issues as possible (I wouldn't use unportable semantics everywhere). But that's my personal opinion.

7
  • What a great answer.. but it's depressing to read the last paragraph. Isn't there a way to get around it? – Student Dec 1 '19 at 22:50
  • @Student Not really. If a standard leaves a behaviour unspecified, then an implementation of the standard has to make a call about what to do in that situation. Two implementations may choose to do things differently, and if you rely an one behaviour, you may just have to be aware that on another system, the same command may fail in a particular way, and program defensively around that. – Kusalananda Dec 1 '19 at 22:54
  • 1
    The GNU utilities are not POSIX compliant because they implement intentional deviations without good reason. The reason why there are more and more "unspecified behavior" notes in the standard is mainly a result of the deviations in the GNU utilities. – schily Dec 1 '19 at 22:57
  • 2
    @schily Hi. I never said they were POSIX compliant. I said they implement the POSIX utilities. How well they adhere to the actual standard, I can't really say much about since I'm not a Linux user to any greater extent. I will however rephrase that sentence a bit to make it clear that it's not an endorsement. – Kusalananda Dec 1 '19 at 23:14
  • 1
    @Student There is no such thing. You can't program without paying attention to details, in any language. Write a piece of code for Linux, put it up on Github, get people to use it, and eventually someone will find incompatibilities and bugs. Then you fix those issues. That's the usual workflow. You could feel safer by just testing your software, obviously, on different Unices, under different conditions, etc. – Kusalananda Dec 2 '19 at 6:35
2

The point of POSIX is to standardize these things, so

  1. bash is fine, but use the POSIX subset.
  2. You lose some efficiency
  3. Read the rest of the POSIX specs, hang around here, and other places such as Usenet
1
  • I wasn't aware of the existence of that community! Thanks – Student Dec 1 '19 at 22:52
2

If you want to write portable scripts on GNU/Linux systems, a good bet is to use the dash shell. This is very portable shell which provides few or no extra capabilities beyond standard. This is the standard /bin/sh for Debian and Ubuntu systems. Red Hat and CentOS use bash as their /bin/sh so they aren't good for testing portability.

5
  • Which one is more standard? dash or "normal" /bin/sh? – Student Dec 2 '19 at 0:47
  • 3
    There is no "normal" /bin/sh. Every system has a shell it calls /bin/sh. Some of them few or no features beyond standard POSIX, some of them support many more features in addition to POSIX. RedHat and CentOS (and Fedora etc.) simply link /bin/sh to /bin/bash so they're the same shell. Debian and Ubuntu (and their derivatives) have /bin/bash for users but install dash as /bin/sh instead of linking it to bash. – MadScientist Dec 2 '19 at 4:26
  • 2
    You've only really mentioned one side of the shell portability issue here. Take any of the /bin/sh shell scripts on Debian or Ubuntu, and try running them on OpenBSD or Solaris. You will find that they most likely fail because even though they are written for sh, they may well use external tools in non-portable ways. In the end, the actual implementation of the shell you use (dash, bash, ksh, zsh) is less important than what utilities you use and how you use them. – Kusalananda Dec 2 '19 at 6:30
  • As a contractor, I work on various client sites, many of whom have locked-down installations for security reasons. It is not much use being a dash guru, or bringing a bunch of great dash scripts with you, if they don't have dash and won't install it for you. – Paul_Pedant Dec 2 '19 at 11:09
  • I guess I've still not made myself clear. What I'm trying to say is that dash is an implementation of POSIX standard sh with no extra non-standard stuff. So if you write your shell scripts so they run in dash, then they should run in any shell that provides compatibility with POSIX sh. That includes bash, ksh, etc. So if your script works in dash you shouldn't need your client to install any particular shell: it will work for any /bin/sh they do have installed. – MadScientist Jun 19 '20 at 19:17
1

Your question seems to be wrong.

The right question would be: "What to do, to end up in portable scripts?"

One main portability issue is programs that implement enhancements and users that use these enhancements, since the enhancements are most likely missing on a different UNIX platform.

Here is a list of frequent causes for portability problems:

  • GNU programs advertise e.g. GNU style long options like --long that are 100% non portable. Never use them in portable scripts.

  • bash implements ksh-style enhancements like [[ -z $HOME ]]. This is a feature that is not in POSIX and only ksh88, ksh93, bash, mksh implement it.

  • bash (unless compiled with the compilation options used by Mac OS and Solaris) implements a non-POSIX echo builtin that is in conflict with the UNIX behavior. Be careful, but also note that:

  • printf is also not a final solution as there are printf implementations that do not deal correctly with nul bytes in string parameters and as different printf implementations have a different feature set.

  • Signal numbers besides 1, 2, 3, 6, 9, 14, and 15 are non-portable. The related signals have different name/number mappings on different UNIX versions.

  • POSIX shell math like echo $(( a + b )) may cause portability issues as different shells come with a different set of supported features. Note that POSIX only requires a limited set of feature.

  • The POSIX shell is typically not in /bin/sh as POSIX does not deal with path names except that is requires dev/tty to be present. If you like to start a POSIX shell, call getconf PATH and use the result as the PATH for your shell, then type sh and you get a POSIX shell.

  • Even all the other utilities that people expect under /bin or /usr/bin typically are not the POSIX variants of the utilities. You need to call:

    PATH=`getconf PATH`
    export PATH
    sh
    

    to get a POSIX shell and this POSIX shell calls the POSIX variants of the utilities.

Many people recommend to use dash for tests, but this is a problem because dash is not POSIX compliant since it does not implement support for multi byte characters.

I recommend to use pbosh which is a compilation variant of bosh. pbosh implements minimal POSIX features that are switched on by default.

1
  • Thanks for a list of things to pay attention. People are right, I'm looking for a way to write portable shell scripts. More importantly, I was wishing to write the most portable shell scripts. Now I learned there's no such way -- this is already a sufficient answer since there are lots of resources out there about writing portable scripts. – Student Dec 2 '19 at 16:29

Not the answer you're looking for? Browse other questions tagged or ask your own question.