5

Sometime, it is not possible to have the luxury of bash on a system, but conditions are easier to make on bash compared to sh or ash, what one should verify to ensure condition won't break with typical "unexpected operator" when rewriting from bash to sh or ash ?

3
  • 1
    It's unclear what the issue is with writing tests in plain sh. Could you give an example?
    – Kusalananda
    Jan 10, 2021 at 7:13
  • arrays not supported
    – alecxs
    Jan 10, 2021 at 10:33
  • 1
    <wiki.ubuntu.com/DashAsBinSh> deals with a number of bashisms. It's not a recent list any longer but still handy when porting from bash to dash. Jan 10, 2021 at 12:41

3 Answers 3

17

I don't agree conditions are easier to make with ((...)) and [[...]] (assuming that's what you're referring to; note that those operators are not specific to bash and come from ksh) than the standard [ or test command. [[ ... ]] and (( ... )) have several problems of their own¹ and which are much worse than those of [.

If your [ fails with an unexpected operator error, it's likely you're not using the shell properly (typically, you forgot to quote an expansion) rather than the [ command.

For how to use [ / test properly and portably, best is to refer to its POSIX specification.

The few ground rules to make it safe are:

  • quote all word expansions ($var, $(cmd), $((arithmetic))) in its arguments. That applies to every command, not just [ and not just to dash. [[ ... ]] and (( ... )) themselves are special constructs with their own specific syntax (varying from shell to shell), where it's not always obvious when things may or may not be quoted.

  • don't pass more than 4 arguments beside the [ and ] ones. That is, don't use the deprecated -o and -a operators, nor ( or ) for grouping. So typically your [ expression should be either:

    • a single argument as in [ "$string" ], though I much prefer the [ -n "$string" ] variant to make it explicit that you check for $string being non-empty.
    • a unary operator (-f, -r, -n...) and its operand, optionally preceded with a ! for negation.
    • a binary operator (=, -gt...) and its 2 operands, optionally preceded by !
    • operands to arithmetic operators must be decimal integer literal constants with an optional sign. dash, like bash accepts leading and trailing whitespace in those operands, but not all [ implementations do.

Check the POSIX standard for the list of portable operators. Note that dash also has a few extensions over the standard (-nt, -ef, -k, -O, <, >² ...)

For pattern matching, use a case construct (case $var in ($pattern)...) instead of if [[ $var = $pattern ]]....

For extended regexp matching, you can use awk:

ere_match() { awk -- 'BEGIN{exit !(ARGV[1] ~ ARGV[2])}' "$1" "$2"; }
if ere_match "$string" "$regex"...

Instead of:

if [[ $string =~ $regex ]]...

For AND/OR, chain several [ invocations with the && or || shell operators (which have equal precedence) and use command groups for grouping, the same you'd use for any other command, not just [.

if
  [ "$mode" = "$mode1" ] || {
    [ -f "$file" ] && [ -r "$file" ]
  }
then...

in place of:

if [[ $mode = "$mode1" || ( -f $file && -r $file ) ]]; then...

Some standard equivalents to some of bash's/dash's/zsh's/ksh's/yash's test/[ non-POSIX operators:

  • -a file 🠖 -e file (but note both return false for symlink to inaccessible files for instance)

  • -k file 🠖

    has_t_bit() (export LC_ALL=C
      ls -Lnd -- "$1" 2> /dev/null | {
        unset -v IFS
        read mode rest &&
          case $mode in
            (*[Tt]) true;;
            (*) false;;
          esac
      }
    )
    
  • -O file ->

    is_mine() (export LC_ALL=C
      ls -Lnd -- "$1" 2> /dev/null | {
        unset -v IFS
        read mode links fuid rest &&
          euid=$(id -u) &&
          [ "$fuid" -eq "$euid" ]
      }
    )
    
  • -G file 🠖

    is_my_group() (export LC_ALL=C
      ls -Lnd -- "$1" 2> /dev/null | {
        unset -v IFS
        read mode links fuid fgid rest &&
          egid=$(id -g) &&
          [ "$fgid" -eq "$egid" ]
      }
    )
    
  • -N file: no equivalent as there's no POSIX API to retrieve a file's modification or access time with full precision.

  • file1 -nt file2 / file1 -ot file2 🠖

    newer() (export LC_ALL=C
      case $1 in ([/.]*) ;; (*) set "./$1" "$2"; esac
      case $2 in ([/.]*) ;; (*) set "$1" "./$2"; esac
      [ -n "$(find -L "$1" -prune -newer "$2" 2> /dev/null)" ]
    )
    older() { newer "$2" "$1"; }
    
    newer file1 file2
    

    beware the behaviour of -nt/-ot varies between implementations if either file is not accessible. Here newer and older return false in those cases.

  • file1 -ef file2 🠖

    same_file() (export LC_ALL=C
      [ "$1" = "$2" ] && [ -e "$1" ] && return
      inode1=$(ls -Lid -- "$1" | awk 'NR == 1 {print $1}') &&
        inode2=$(ls -Lid -- "$2" | awk 'NR == 1 {print $1}') &&
        [ -n "$inode1" ] && [ "$inode1" -eq "$inode2" ] &&
        dev1=$(df -P -- "$1" | awk 'NR == 2 {print $1}') &&
        dev2=$(df -P -- "$2" | awk 'NR == 2 {print $1}') &&
        [ -n "$dev1" ] && [ "$dev1" = "$dev2" ]
    ) 2> /dev/null
    
    same_file file1 file2
    

    (assuming filesystem sources in the df output don't contain whitespace; the filesystem comparison also doesn't work properly if the operands are actually devices with filesystems mounted on them).

  • -v varname 🠖 [ -n "${varname+set}" ]

  • -o optname 🠖 case $- in (*"$single_letter_opt_name"*).... For long option name, I can't think of a standard equivalent.

  • -R varname 🠖

    is_readonly() (export LC_ALL=C
      _varname=$1
      is_readonly_helper() {
        [ "${1%%=*}" = "$_varname" ] && exit
      }
      eval "$(
        readonly -p |
          sed 's/^[[:blank:]]*readonly/is_readonly_helper/'
      )"
      exit 1
    )
    

    Though beware it's dangerous as sed may be affected by a LINE_MAX limit while variables have no length limit.

    Also note that in ksh93, -R name is to check whether name is a nameref variable.

  • "$a" == "$b" 🠖 "$a" = "$b"

  • "$a" '<' "$b", "$a" '>' "$b" 🠖

    collate() {
      awk -- 'BEGIN{exit !(ARGV[1] "" '"$2"' ARGV[2] "")}' "$1" "$3"
    }
    collate "$a" '<' "$b"
    collate "$a" '>' "$b"
    

    (that collate also supports <=, >=, ==, !=).

    Whether [, or awk's < compares strings using byte-to-byte comparison or the locale's collation order depends on the implementation. dash has not been internationalised, so it only works with byte values. For yash's [, it's based on locale collation. For bash, [[ $a < $b ]] works with locale collation while [ "$a" '<' "$b" ] works with byte value. POSIX requires awk's < to use the locale's collation order but some awk implementations like mawk have not been internationalised so work with byte values. To force byte-value comparison, fix the locale to C.

    For awk's == and != operators, POSIX used to require collation be used, but few implementations do and the POSIX specification now leaves it unspecified. ['s = and != always do byte-to-byte comparison, but see yash's === / !== below.

  • "$a" '=~' "$b" 🠖 see above (while in bash/ksh, the =~ operator is available only in [[...]], that's not the case of zsh or yash whose [ supports it).

  • "$string" -pcre-match "$pcre" 🠖 no equivalent as PCREs are not specified by POSIX.

  • "$a" === "$b" / "$a" !== "$b" (strcoll() comparison) 🠖 expr "z $a" = "z $b" > /dev/null / expr "z $a" != "z $b" (also note that some awk implementations ==/!= operator also use strcoll() for comparison).

  • -o "?$option_name" (is valid option name) 🠖 no standard equivalent that I can think of as the output format of set -o or set +o is unspecified.

  • "$version1" -veq "$version2" (and -vne/-vgt/-vge/-vlt/-vle to compare version numbers) 🠖

    version_compare() {
      awk -- '
        function pad(s,   r) {
          r = ""
          while (match(s, /[0123456789]+/)) {
            r = r substr(s, 1, RSTART - 1) \
                sprintf("%020u", substr(s, RSTART, RLENGTH))
            s = substr(s, RSTART + RLENGTH)
          }
          return r s
        }
        BEGIN {exit !(pad(ARGV[1]) '"$2"' pad(ARGV[2]))}' "$1" "$3"
    }
    

    used as version_compare "$v1" '<' "$v2" (or <=, ==, !=, >=, >), using the same meaning as yash's operators (or zsh's numeric order) do, and assuming none of the numbers have more than 20 digits.

  • The bosh shell's [ also has -D file and -P file to test whether a file is a door or event port. Those are types of files that are specific to Solaris, so not covered by POSIX, but you could always define a:

    is_of_type() (export LC_ALL=C
      case "$2" in
        ([./]*) file="$2";;
            (*) file="./$2"
      esac
      [ -n "$(find -H "$file" -prune -type "$1")" ]
    )
    

    Then, you can do is_of_type D "$file" which even though not POSIX would likely work on Solaris where doors are relevant, or the equivalent on other systems and their own specific types of files.


¹ see for instance the command injection vulnerabilities in [[...]]'s arithmetic operators and ((...)), the mess with the =~ operator in bash3.2+, ksh93 or yash.

² some of which are quite widespread among [ implementations and might be added in a future version of the POSIX standard

0
10

Stéphane Chazelas has an excellent answer on pointing out differences in syntax between narrowly POSIX-compliant shells and other shells, and other pitfalls. This answer will take a different approach.

As a human it can be difficult to take a script written for bash, ksh, or some other shell with their non-POSIX extensions and safely translate the entire script into a script using only POSIX syntax. It is easy to accidentally miss something, and then the script might fail on you, and possibly at a very bad time.

There is a wonderful open source tool I have been using for some time now, called shellcheck. What shellcheck does is it works as a linter for shell scripts. If you specify a shebang like #!/usr/bin/env sh or use the --shell sh option, shellcheck will automatically alert you of any syntax that is invalid for the specified shell.

Shellcheck also has a very nice wiki where every linting error or warning is documented on its own page, with an explanation for what the error or warning means and suggestions for correct/safer code.

If you choose to install shellcheck, I recommend you install that latest stable release from the releases page on GitHub rather than using a package manager, since in my experience package managers have old versions of shellcheck.

6
  • Yes, it's very useful. It's a pity it misses common mistakes like the missing quotes in [ $# -gt 2 ]. It also doesn't pick up non-standard extensions like [ -k file ], Jan 10, 2021 at 19:28
  • Kept Stéphane's reply as answer, but upvoted yours and hope people will keep upvoting it, good content there !
    – Zulgrib
    Jan 10, 2021 at 21:48
  • In all honesty, Stéphane's answer really is the better answer. Among other things, it contains functions for replacing several non-POSIX operators. I just wanted to point out a tool exists that can help a human track down syntax in their shell scripts that does not comply with the syntax of the shell they are writing the script for. Jan 10, 2021 at 23:37
  • 3
    I created a GitHub issue for the [ -k file ] case and for the lack of warnings for other non-POSIX operators: github.com/koalaman/shellcheck/issues/2125 Jan 11, 2021 at 0:02
  • 3
    @ShaneBishop see unix.stackexchange.com/a/171347/117549 for fun with IFS and $#
    – Jeff Schaller
    Jan 11, 2021 at 3:32
-2

I'd say the most important thing to avoid is [[ ... ]]. Arithmetic in between (( ... )) probably doesn't work either.

Make it a habit to put variables in quotes to avoid errors when a variable is not set, such as "$var".

All operators, arithmetic or logical, are words with dashes such as -eq or -a instead of = or &&.

Also use shellcheck.net for syntax checks. #!/bin/sh tells shellcheck that you want to use the POSIX shell, which is not quite the Bourne shell but close, I think. There should be a way to test Bourne shell scripts, too.

4
  • 4
    "Probably doesn't work"? Consider looking that up and make your statement a bit more certain. The issue with unquoted variables has to do with splitting and globbing. The fact that an unset variable causes issues if unquoted in a test is merely a consequence of this. What's wrong with using =?
    – Kusalananda
    Jan 10, 2021 at 7:17
  • I believe (( ... )) is standardized in POSIX, but I haven't found a description of Bourne shell features and doubt that it works in the Bourne shell. I'll leave it up to OP to look it up or otherwise confirm this. You are right that the issue with unquoted variables has to do with something, and my point is that it is an issue. Again you are right that test allows string comparison with =. Not so with arithmetic comparison, thoug, where -eq is required. Jan 10, 2021 at 8:38
  • (( ... )) (arithmetic evaluation) is an extension to the POSIX standard. BTW, the standard is available here. You text currently says that "All operators ... are words with dashes" and that = (and &&) should not be used.
    – Kusalananda
    Jan 10, 2021 at 9:09
  • 1
    +1 for kiss answer and no wot (mods who deleted this comment, kiss - keep it simple stupid, wot - wall of text. this is coders language and no slang) while the accepted answer is more detailed, this answer was predecessor. it already contains basic differences, bash`s double brackets, quoted vars, operators. furthermore shellcheck is mentioned. i see no reason to downvote just for nitpicking
    – alecxs
    Jan 11, 2021 at 11:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.