Why zsh and ksh93 chose to be non-compliant in pattern matching?

Question

POSIX documentation for pattern matching said that:

An ordinary character is a pattern that shall match itself. It can be any character in the supported character set except for NUL, those special shell characters in Quoting that require quoting, and the following three special pattern characters. Matching shall be based on the bit pattern used for encoding the character, not on the graphic representation of the character. If any character (ordinary, shell special, or pattern special) is quoted, that pattern shall match the character itself. The shell special characters always require quoting.

As I understand, the pattern ["!"a] will match any of ! and a. That's also the behavior in most shells I tried, except zsh and ksh93:

$ for shell in /bin/*[^c]sh; do
  printf '=%-17s=\n' "$shell"
  "$shell" -c 'case a in ["!"a]) echo 1;; esac'
done
=/bin/ash         =
1
=/bin/bash        =
1
=/bin/dash        =
1
=/bin/heirloom-sh =
1
=/bin/ksh         =
=/bin/lksh        =
1
=/bin/mksh        =
1
=/bin/pdksh       =
1
=/bin/posh        =
1
=/bin/schily-osh  =
1
=/bin/schily-sh   =
1
=/bin/yash        =
1
=/bin/zsh         =

zsh and ksh93 seem to treat ["!"a] the same as [!a], which match any character except a:

$ for shell in ksh93 zsh; do
  printf '=%-6s=\n' "$shell"
  "$shell" -c 'case b in ["!"a]) echo 1;; esac'
done
=ksh93 =
1
=zsh   =
1

Is there any reason (historical, development, ...) for zsh and ksh93 behave like that?

---

zsh does the same thing in both ksh and sh emulation.

busybox sh, Solaris /usr/xpg4/bin/sh and FreeBSD sh also behave like POSIX documentation.

---

ksh88 also behave like most other shells, the behavior changed between kssh88 and ksh93:

$ ksh88 -c 'case a in ["!a"]) echo yes; esac'
yes
$ ksh88 -c 'case b in ["a-c"]) echo yes; esac' 
$

Answer 1

The passage you quote does not mean what you say it means.

Patterns Matching a Single Character

(…) An ordinary character is a pattern that shall match itself. (…) If any character (ordinary, shell special, or pattern special) is quoted, that pattern shall match the character itself.

All of this applies only to characters that stand for themselves in a pattern. This does not apply to characters that appear in a context other than that where a pattern character is expected. In particular, it does not apply inside a bracket expression. The syntax of bracket expressions is described under the entry for [:

If an open bracket introduces a bracket expression as in XBD RE Bracket Expression, (…)

(I omitted the bit about ! vs ^ for complementation.) The description of RE bracket expressions doesn't say anything about quoting (which is unsurprising since it's about bracket expressions in general, not about bracket expressions in a pattern in a shell script).

Going by a strict interpretation of POSIX.1-2008, it isn't clear what the pattern ["!"a] should match. One interpretation is that it should match any of the characters ", ! or a: the character " has no special meaning inside a bracket expression. I can't find anything in the specification that would invalidate this interpretation. Another interpretation is that " retains its quoting behavior, but that means that the content of the bracket expression is !a, and since there is no particular treatment of quoted characters inside bracket expressions, the set is all-but-a. I can't find any support for your interpretation (and the behavior of dash, bash and other shells) in the POSIX specification. It makes sense, sure, but it isn't what the text says.

It would make sense for a future version of POSIX to mandate your interpretation, by adding some wording to this effect. For example, the description of [ could be changed to

If an open bracket introduces a bracket expression as in XBD RE Bracket Expression, except that the \ character ('!') shall replace the \ character ('^') in its role in a non-matching list in the regular expression notation, it shall introduce a pattern bracket expression, and that any character that is quoted shall stand for itself as an element of the enclosing bracket expression, collating element or class expression. A bracket expression starting with an unquoted \ character produces unspecified results. Otherwise, '[' shall match the character itself.

Given that POSIX is mostly descriptive rather than normative, I'd expect such a change that breaks ksh (usually the reference shell) to only be included in a major update of the standard, and any defect on the existing version to instead allow at least the existing different interpretations.

Answer 2

That's a bug in zsh, which was reported in this discussion [BUG] quoting within bracket patterns has no effect:

case b in
  ( ['a-c'] ) echo 'false match' ;;
  ( [a-c] )   echo 'correct match' ;;
esac

will print false match instead of correct match.

The fix is scheduled to be released with zsh version 5.3.

Answer 3

What you are reading is applicable only to simple characters. Not characters inside a Bracket expression.

In fact, that is clearly stated a little ahead:

When unquoted and outside a bracket expression, the following three characters shall have special meaning in the specification of patterns:

? A <question-mark> ...
* An <asterisk> ...
[ If an open bracket introduces a bracket expression ...

What you need to read for a Bracket Expression is here.

According to the spec; Inside a "Bracket Expression", there is no concept of quoting (for patterns).

However most shell do remove quotes on any string, even if that string is inside a "Bracket Expression". That is why a ["!"a] becomes [!a] for the command case.

However, the shell retains a knowledge of that the string was quoted for most shell and so the negation does not take effect (in opposition to the spec not having a concept of quoting inside "Bracket Expression").

In ksh and zsh that knowledge is not used to evaluate the pattern.

Why does that happen?, I believe that those are just mistakes.

---

However, ksh and zsh act differently than most shells.

Using this code (the case are repeated to test all values in all shells):

whichsh="`ps -o pid,args| awk '$1=='"$$"'{print $2}'`"
[ ${whichsh##*/} = zsh  ] && setopt GLOB_SUBST
[ ${whichsh##*/} = zsh4 ] && setopt GLOB_SUBST

a="$1"; printf '%s\t' "testing $a"

case $a in ['!a'])    printf 1 ;; esac
case $a in ["!a"])    printf 2 ;; esac
case $a in ['!'a])    printf 3 ;; esac
case $a in ["!"a])    printf 4 ;; esac
case $a in [\"!\"a])  printf 5 ;; esac
case $a in [!a])      printf 6 ;; esac
printf "\t --"

t1="['!a']";t2='["!a"]';t3="['!'a]";t4='["!"a]';t5='[\"!\"a]'
case $a in $t1)     printf 1 ;; esac
case $a in $t2)     printf 2 ;; esac
case $a in $t3)     printf 3 ;; esac
case $a in $t4)     printf 4 ;; esac
case $a in $t5)     printf 5 ;; esac
case $a in [!a])    printf 6 ;; esac
echo

For a test with "a", ./script.sh a, the results are:

/bin/dash       : testing a     12345    --12345
/bin/sh         : testing a     12345    --12345
/bin/b43sh      : testing a     12345    --12345
/bin/b44sh      : testing a     12345    --12345
/bin/bash       : testing a     12345    --12345
/bin/ksh        : testing a     5        --12345
/bin/ksh93      : testing a     5        --12345
/bin/lksh       : testing a     12345    --12345
/bin/mksh       : testing a     12345    --12345
/bin/zsh        : testing a     5        --12345
/bin/zsh4       : testing a     5        --12345

And a test for "b", ./script.sh b, the resuts are:

/bin/dash       : testing b     6        --6
/bin/sh         : testing b     6        --6
/bin/b43sh      : testing b     6        --6
/bin/b44sh      : testing b     6        --6
/bin/bash       : testing b     6        --6
/bin/ksh        : testing b     12346    --6
/bin/ksh93      : testing b     12346    --6
/bin/lksh       : testing b     6        --6
/bin/mksh       : testing b     6        --6
/bin/zsh        : testing b     12346    --6
/bin/zsh4       : testing b     12346    --6

When the test patterns have quotes inside variables, the quotes are not remove and affect the result. When the quotes are used directly in the pattern, ksh and zsh remove the quotes and evaluate [!a].

What all shells should do is keep the quotes inside the brackets as part of the tested characters.

To get quotes "quoted", ksh and zsh need a pattern like [\"!\"a]. Then both match a, ! and ".