6

POSIX documentation for pattern matching said that:

An ordinary character is a pattern that shall match itself. It can be any character in the supported character set except for NUL, those special shell characters in Quoting that require quoting, and the following three special pattern characters. Matching shall be based on the bit pattern used for encoding the character, not on the graphic representation of the character. If any character (ordinary, shell special, or pattern special) is quoted, that pattern shall match the character itself. The shell special characters always require quoting.

As I understand, the pattern ["!"a] will match any of ! and a. That's also the behavior in most shells I tried, except zsh and ksh93:

$ for shell in /bin/*[^c]sh; do
  printf '=%-17s=\n' "$shell"
  "$shell" -c 'case a in ["!"a]) echo 1;; esac'
done
=/bin/ash         =
1
=/bin/bash        =
1
=/bin/dash        =
1
=/bin/heirloom-sh =
1
=/bin/ksh         =
=/bin/lksh        =
1
=/bin/mksh        =
1
=/bin/pdksh       =
1
=/bin/posh        =
1
=/bin/schily-osh  =
1
=/bin/schily-sh   =
1
=/bin/yash        =
1
=/bin/zsh         =

zsh and ksh93 seem to treat ["!"a] the same as [!a], which match any character except a:

$ for shell in ksh93 zsh; do
  printf '=%-6s=\n' "$shell"
  "$shell" -c 'case b in ["!"a]) echo 1;; esac'
done
=ksh93 =
1
=zsh   =
1

Is there any reason (historical, development, ...) for zsh and ksh93 behave like that?


zsh does the same thing in both ksh and sh emulation.

busybox sh, Solaris /usr/xpg4/bin/sh and FreeBSD sh also behave like POSIX documentation.


ksh88 also behave like most other shells, the behavior changed between kssh88 and ksh93:

$ ksh88 -c 'case a in ["!a"]) echo yes; esac'
yes
$ ksh88 -c 'case b in ["a-c"]) echo yes; esac' 
$
  • The problem of using only positive tests. The second test only tells me ! is matched, not the broader "any character except a". – muru Feb 24 '16 at 9:13
  • 1
    @muru: Sorry, it's b, not !, it's copy-paste mistake. – cuonglm Feb 24 '16 at 9:20
  • 2
    There's already been some fixes related to quoting and [] recently in zsh, not enough it would seem. You may want to bring the discussion to the zsh mailing list. Check the archive of the ml. – Stéphane Chazelas Feb 24 '16 at 11:37
  • @StéphaneChazelas: Yes, I will. How about ksh, it's the one actually surprise me a lot. I don't have ksh88 for testing. But I guess it will be compliant. Can you help me checking it? – cuonglm Feb 24 '16 at 11:43
  • The POSIX certified ksh88 from Solaris behaves like most other shells. POSIX requires the case statement to use the glob() pattern matching method. The way I understand the standard makes me believe that ksh93 is wrong. – schily Feb 29 '16 at 19:51
5

The passage you quote does not mean what you say it means.

Patterns Matching a Single Character

(…) An ordinary character is a pattern that shall match itself. (…) If any character (ordinary, shell special, or pattern special) is quoted, that pattern shall match the character itself.

All of this applies only to characters that stand for themselves in a pattern. This does not apply to characters that appear in a context other than that where a pattern character is expected. In particular, it does not apply inside a bracket expression. The syntax of bracket expressions is described under the entry for [:

If an open bracket introduces a bracket expression as in XBD RE Bracket Expression, (…)

(I omitted the bit about ! vs ^ for complementation.) The description of RE bracket expressions doesn't say anything about quoting (which is unsurprising since it's about bracket expressions in general, not about bracket expressions in a pattern in a shell script).

Going by a strict interpretation of POSIX.1-2008, it isn't clear what the pattern ["!"a] should match. One interpretation is that it should match any of the characters ", ! or a: the character " has no special meaning inside a bracket expression. I can't find anything in the specification that would invalidate this interpretation. Another interpretation is that " retains its quoting behavior, but that means that the content of the bracket expression is !a, and since there is no particular treatment of quoted characters inside bracket expressions, the set is all-but-a. I can't find any support for your interpretation (and the behavior of dash, bash and other shells) in the POSIX specification. It makes sense, sure, but it isn't what the text says.

It would make sense for a future version of POSIX to mandate your interpretation, by adding some wording to this effect. For example, the description of [ could be changed to

If an open bracket introduces a bracket expression as in XBD RE Bracket Expression, except that the \ character ('!') shall replace the \ character ('^') in its role in a non-matching list in the regular expression notation, it shall introduce a pattern bracket expression, and that any character that is quoted shall stand for itself as an element of the enclosing bracket expression, collating element or class expression. A bracket expression starting with an unquoted \ character produces unspecified results. Otherwise, '[' shall match the character itself.

Given that POSIX is mostly descriptive rather than normative, I'd expect such a change that breaks ksh (usually the reference shell) to only be included in a major update of the standard, and any defect on the existing version to instead allow at least the existing different interpretations.

  • @cuonglm No, the passage you quoted does not say that a character in a bracket expression stands for itself is quoted. It says that only for a character in a pattern. This isn't the same thing. If that sentence applied to characters inside a bracket expression, it would mean that ["!"a] would match !, since ! would “match itself” — and that wouldn't make any sense. Don't confuse what you would like the sentence to mean and what it actually means. That's often a problem when reading standards. – Gilles Feb 28 '16 at 23:09
  • @cuonglm What in that thread is relevant? It's about case, and I don't see any proposed change that would affect the meaning of patterns. All the wording about quoting is that it causes characters to match themselves — but that does not apply to characters in a bracket expression, which do not match themselves, but are part of a set that ends up matching themselves or (for [!…]) not matching themselves, or could even be endpoints in a range or a class etc. (Do you think [[:alpha\:]] and [[:alpha"":]] should match b? Dash doesn't treat those two in the same way!) – Gilles Feb 29 '16 at 8:43
  • It's clearer for me now. Thank you for pointing out my misleading in interpretation – cuonglm Feb 29 '16 at 17:49
2

That's a bug in zsh, which was reported in this discussion [BUG] quoting within bracket patterns has no effect:

case b in
  ( ['a-c'] ) echo 'false match' ;;
  ( [a-c] )   echo 'correct match' ;;
esac

will print false match instead of correct match.

The fix is scheduled to be released with zsh version 5.3.

1

What you are reading is applicable only to simple characters. Not characters inside a Bracket expression.

In fact, that is clearly stated a little ahead:

When unquoted and outside a bracket expression, the following three characters shall have special meaning in the specification of patterns:

? A <question-mark> ...
* An <asterisk> ...
[ If an open bracket introduces a bracket expression ...

What you need to read for a Bracket Expression is here.

According to the spec; Inside a "Bracket Expression", there is no concept of quoting (for patterns).

However most shell do remove quotes on any string, even if that string is inside a "Bracket Expression". That is why a ["!"a] becomes [!a] for the command case.

However, the shell retains a knowledge of that the string was quoted for most shell and so the negation does not take effect (in opposition to the spec not having a concept of quoting inside "Bracket Expression").

In ksh and zsh that knowledge is not used to evaluate the pattern.

Why does that happen?, I believe that those are just mistakes.


However, ksh and zsh act differently than most shells.

Using this code (the case are repeated to test all values in all shells):

whichsh="`ps -o pid,args| awk '$1=='"$$"'{print $2}'`"
[ ${whichsh##*/} = zsh  ] && setopt GLOB_SUBST
[ ${whichsh##*/} = zsh4 ] && setopt GLOB_SUBST

a="$1"; printf '%s\t' "testing $a"

case $a in ['!a'])    printf 1 ;; esac
case $a in ["!a"])    printf 2 ;; esac
case $a in ['!'a])    printf 3 ;; esac
case $a in ["!"a])    printf 4 ;; esac
case $a in [\"!\"a])  printf 5 ;; esac
case $a in [!a])      printf 6 ;; esac
printf "\t --"

t1="['!a']";t2='["!a"]';t3="['!'a]";t4='["!"a]';t5='[\"!\"a]'
case $a in $t1)     printf 1 ;; esac
case $a in $t2)     printf 2 ;; esac
case $a in $t3)     printf 3 ;; esac
case $a in $t4)     printf 4 ;; esac
case $a in $t5)     printf 5 ;; esac
case $a in [!a])    printf 6 ;; esac
echo

For a test with "a", ./script.sh a, the results are:

/bin/dash       : testing a     12345    --12345
/bin/sh         : testing a     12345    --12345
/bin/b43sh      : testing a     12345    --12345
/bin/b44sh      : testing a     12345    --12345
/bin/bash       : testing a     12345    --12345
/bin/ksh        : testing a     5        --12345
/bin/ksh93      : testing a     5        --12345
/bin/lksh       : testing a     12345    --12345
/bin/mksh       : testing a     12345    --12345
/bin/zsh        : testing a     5        --12345
/bin/zsh4       : testing a     5        --12345

And a test for "b", ./script.sh b, the resuts are:

/bin/dash       : testing b     6        --6
/bin/sh         : testing b     6        --6
/bin/b43sh      : testing b     6        --6
/bin/b44sh      : testing b     6        --6
/bin/bash       : testing b     6        --6
/bin/ksh        : testing b     12346    --6
/bin/ksh93      : testing b     12346    --6
/bin/lksh       : testing b     6        --6
/bin/mksh       : testing b     6        --6
/bin/zsh        : testing b     12346    --6
/bin/zsh4       : testing b     12346    --6

When the test patterns have quotes inside variables, the quotes are not remove and affect the result. When the quotes are used directly in the pattern, ksh and zsh remove the quotes and evaluate [!a].

What all shells should do is keep the quotes inside the brackets as part of the tested characters.

To get quotes "quoted", ksh and zsh need a pattern like [\"!\"a]. Then both match a, ! and ".

  • @Gilles could you take a look at this answer? Do you have an opinion of it? Thanks. – user79743 Feb 29 '16 at 15:06
  • With zsh, you need $~var (or enable the globsubst option like in sh emulation) for variables to be taken as patterns. In any case, where patterns are in variables, that's quite a different matter as quotes are not removed after parameter expansion (though backslash still have a special meaning within bracket expressions in bash and dash). – Stéphane Chazelas Feb 29 '16 at 17:37
  • @StéphaneChazelas Added an specific setopt GLOB_SUBST for zsh expansion of patterns (and the effect that has in the result). What I find very interesting is that with the modifications proposed to zsh a variable quoted or not work exactly the same (inside "Bracket Expressions" that is): [$cset] and ["$cset"] work the same way in zsh. – user79743 Feb 29 '16 at 20:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.