Why is bash removing other digits?

Question

On this (It is not intended to be a range, but an explicit list):

$ a='0123456789 ٠١٢٣٤٥٦٧٨٩ ۰۱۲۳۴۵۶۷۸۹ ߀߁߂߃߄߅߆߇߈߉ ०१२३४५६७८९'
$ echo "${a//[0123456789]}"
  ۰۱۲۳۴۵۶۷۸۹ ߀߁߂߃߄߅߆߇߈߉ ०१२३४५६७८९

Bash is incorrectly (IMO) removing the digits ٠١٢٣٤٥٦٧٨٩ (the second group).

---

The characters are all different (hand formatted):

$ for c in $(echo "$a" | grep -o .); do printf '\\U%04x ' "'$c"; done; echo
\U0030 \U0031 \U0032 \U0033 \U0034 \U0035 \U0036 \U0037 \U0038 \U0039
\U0660 \U0661 \U0662 \U0663 \U0664 \U0665 \U0666 \U0667 \U0668 \U0669
\U06f0 \U06f1 \U06f2 \U06f3 \U06f4 \U06f5 \U06f6 \U06f7 \U06f8 \U06f9
\U07c0 \U07c1 \U07c2 \U07c3 \U07c4 \U07c5 \U07c6 \U07c7 \U07c8 \U07c9
\U0966 \U0967 \U0968 \U0969 \U096a \U096b \U096c \U096d \U096e \U096f

Which correspond to:

123456789    # Hindu-Arabic Arabic numerals
٠١٢٣٤٥٦٧٨٩   # ARABIC-INDIC
۰۱۲۳۴۵۶۷۸۹   # EXTENDED ARABIC-INDIC/PERSIAN
߀߁߂߃߄߅߆߇߈߉  # NKO DIGIT
०१२३४५६७८९   # DEVANAGARI

To ensure there are no problems with pasting from this website, it is also possible to produce this Unicode content into the a variable using the Unicode escapes:

a=$(echo -e '\u0030\u0031\u0032\u0033\u0034\u0035\u0036\u0037\u0038\u0039 \u0660\u0661\u0662\u0663\u0664\u0665\u0666\u0667\u0668\u0669 \u06f0\u06f1\u06f2\u06f3\u06f4\u06f5\u06f6\u06f7\u06f8\u06f9 \u07c0\u07c1\u07c2\u07c3\u07c4\u07c5\u07c6\u07c7\u07c8\u07c9 \u0966\u0967\u0968\u0969\u096a\u096b\u096c\u096d\u096e\u096f')

Or using the $'...' strings which accept escapes directly:

a=$'\u0030\u0031\u0032\u0033\u0034\u0035\u0036\u0037\u0038\u0039 \u0660\u0661\u0662\u0663\u0664\u0665\u0666\u0667\u0668\u0669 \u06f0\u06f1\u06f2\u06f3\u06f4\u06f5\u06f6\u06f7\u06f8\u06f9 \u07c0\u07c1\u07c2\u07c3\u07c4\u07c5\u07c6\u07c7\u07c8\u07c9 \u0966\u0967\u0968\u0969\u096a\u096b\u096c\u096d\u096e\u096f'

---

Other shells do not work as bash (hand formatted):

$ for sh in zsh ksh lksh mksh bash; do $sh -c 'a="0123456789 ٠١٢٣٤٥٦٧٨٩ ۰۱۲۳۴۵۶۷۸۹ ߀߁߂߃߄߅߆߇߈߉ ०१२३४५६७८९"; echo "$0 : ${a//[0123456789]}" $sh'; done
zsh  :  ٠١٢٣٤٥٦٧٨٩ ۰۱۲۳۴۵۶۷۸۹ ߀߁߂߃߄߅߆߇߈߉ ०१२३४५६७८९
ksh  :  ٠١٢٣٤٥٦٧٨٩ ۰۱۲۳۴۵۶۷۸۹ ߀߁߂߃߄߅߆߇߈߉ ०१२३४५६७८९
lksh :  ٠١٢٣٤٥٦٧٨٩ ۰۱۲۳۴۵۶۷۸۹ ߀߁߂߃߄߅߆߇߈߉ ०१२३४५६७८९
mksh :  ٠١٢٣٤٥٦٧٨٩ ۰۱۲۳۴۵۶۷۸۹ ߀߁߂߃߄߅߆߇߈߉ ०१२३४५६७८९
bash :   ۰۱۲۳۴۵۶۷۸۹ ߀߁߂߃߄߅߆߇߈߉ ०१२३४५६७८९

---

The bash sort order is:

$ mkdir test1; cd test1; IFS=$' \t\n'
$ touch $(echo "$a" | grep -o .)
$ printf '%s' *; echo
߃߇߆߁߂߅߉߄߀߈0٠०۰1١१۱٢2२۲3٣३۳٤4४۴٥5५۵٦6६۶7٧७۷8٨८۸٩9९۹

$ locale
LANG=en_US.utf8
LANGUAGE=
LC_CTYPE="en_US.utf8"
LC_NUMERIC="en_US.utf8"
LC_TIME="en_US.utf8"
LC_COLLATE="en_US.utf8"
LC_MONETARY="en_US.utf8"
LC_MESSAGES="en_US.utf8"
LC_PAPER="en_US.utf8"
LC_NAME="en_US.utf8"
LC_ADDRESS="en_US.utf8"
LC_TELEPHONE="en_US.utf8"
LC_MEASUREMENT="en_US.utf8"
LC_IDENTIFICATION="en_US.utf8"
LC_ALL=

It doesn't seem to be applying the sort order to remove characters.

It shouldn't anyway (IMO) as the characters are being explicitly listed.

So: Why?

---

Using bash 4.4.12 here. But it fails also with 3.0, 3.2, 4.0, 4.1, 4.4.23, 5.0 but not with 2.0.1 nor 2.0.5. It seems that a change in 3.0 caused the issue.

Answer 1

I managed to reproduce this problem on Ubuntu 17.10 (glibc 2.26) and on Ubuntu 18.04 (glibc 2.27), but it seems to be fixed on Ubuntu 18.10 (glibc 2.28)

The problem is with the localedata, more specifically the LC_COLLATE data for en_US.utf8 (actually, that collation data comes from an ISO 14651 file which is included in most locales, so it probably affects all other utf8 locales as well.)

The localedata comes from glibc and the bug seems to be present there (though distros customize that data fairly heavily, so it's possible other distros with glibc <2.28 might not have the issue.)

In fact, the glibc 2.28 announcement starts listing new features with:

The localization data for ISO 14651 is updated to match the 2016 Edition 4 release of the standard, this matches data provided by Unicode 9.0.0. This update introduces significant improvements to the collation of Unicode characters.

Looking at the commits, it's a huge overhaul on the localedata, so that's probably what fixed the bug!

In short, the issue with collation of these two symbols (U0030, which is '0', and U0660, which is the Arabic-Indic zero '٠') is that they sort exactly the same, when compared using strcoll(3), which can be demonstrated with this short test using sort (which uses strcoll under the hood):

ubuntu-18.04$ { echo 0; echo -e '\u0660'; echo 0; } | sort
0
٠
0

And on glibc 2.28:

ubuntu-18.10$ { echo 0; echo -e '\u0660'; echo 0; } | sort
0
0
٠

As you can see, on the older glibc, it's not reordering the Arabic-Indic zero '٠', neither before nor after the '0', which proves they collate the same.

Looking at the glibc sources, we can understand why the problem happens.

In the glibc 2.27 sources for ISO 14651, the following definitions can be found:

<U0030> <0>;<BAS>;<MIN>;IGNORE # 171 0
<U0660> <0>;<BAS>;<MIN>;IGNORE
<U06F0> <0>;<PCL>;<MIN>;IGNORE
<U0966> <0>;"<BAS><NUM>";"<MIN><MIN>";IGNORE

So both '0' (\u0030) and '٠' (\u0660) expand to the exact same sequence (<0>;<BAS>;<MIN>;IGNORE) which means that strcoll will treat them the same. (This also explains why the other characters such as \u06f0 and \u0966 are not affected, since their expansion is different.)

Looking at the glibc 2.28 sources for ISO 14651, the following definitions are now found:

<U0030> <S0030>;<BASE>;<MIN>;<U0030> % DIGIT ZERO
<U0660> <S0030>;<BASE>;<MIN>;<U0660> % ARABIC-INDIC DIGIT ZERO
<U06F0> <S0030>;<BASE>;<MIN>;<U06F0> % EXTENDED ARABIC-INDIC DIGIT ZERO
<U07C0> <S0030>;<BASE>;<MIN>;<U07C0> % NKO DIGIT ZERO
<U0966> <S0030>;<BASE>;<MIN>;<U0966> % DEVANAGARI DIGIT ZERO

The fourth field is now always filled with the code point itself, which means they will have a defined sort order, even if the first few fields match. While the change for <U0660> was not introduced in this particular commit, its description explains the idea:

[...] putting the code point of the character into the fourth level instead of “IGNORE”. Without that change, all such characters would compare equal which would make a wcscoll test case fail. It is better to have a clearly defined sort order even for characters like this so it is good to use the code point as a tie-break.

• localedata/locales/iso14651_t1_common: Use the code point of a character in the fourth collation level instead of IGNORE for all entries which have IGNORE on all 4 levels.

So hopefully this explains the bug with localedata in glibc <2.28 and the fix in glibc 2.28.

---

Regarding bash, if you look at the source code, you'll see that it handles a single character (0) in a bracket expression ([0]) the same as if it was a range with the character as both start and end ([0-0]):

cstart = cend = FOLD (cstart);

Then later it compares the current character with that range using RANGECMP:

if (RANGECMP (test, cstart, forcecoll) >= 0 && RANGECMP (test, cend, forcecoll) <= 0)
  goto matched;

And then RANGECMP (defined to rangecmp_wc in multi-byte mode) uses wcscoll(3) (which is the multi-byte version of strcoll):

return (wcscoll (s1, s2));

The fact that bash uses a range comparison for a single character (as a shortcut, to share a bit of the code with handling of ranges) makes it so that it accepts all characters that sort the same as well as the original character.

Other shells probably don't have this problem because they do a straight comparison if a range is not involved.

The reason why this issue started appearing on bash 3.0 is that bash 3.0 introduced support for multi-byte (Unicode), which ended up refactoring all this code and probably using locale-aware comparisons, which are connected to the issue.

UPDATE: This issue was reported as a bug to the bash project by @Isaac.

---

WORKAROUND: If upgrading to a distro that uses glibc 2.28 is unfeasible, a possible workaround is to use LC_COLLATE=C.utf8 or POSIX.utf8 which define a "trivial" sort order where no codepoints will sort the same. Considering the issue is with collation, setting LC_COLLATE only is enough. Testing this workaround on Ubuntu 17.10 and 18.04 showed it was enough to fix this problem.