Why does [A-Z] match lowercase letters in bash?

Question

In all shells I am aware of, rm [A-Z]* removes all files that start with an uppercase letter, but with bash this removes all files that start with a letter.

As this problem exists on Linux and Solaris with bash-3 and bash-4, it cannot be a bug caused by a buggy pattern matcher in libc or a miss-configured locale definition.

Is this strange and risky behavior intended or is this just a bug that exists unfixed since many years?

Answer 1

Note that when using range expressions like [a-z], letters of the other case may be included, depending on the setting of LC_COLLATE.

LC_COLLATE is a variable which determines the collation order used when sorting the results of pathname expansion, and determines the behavior of range expressions, equivalence classes, and collating sequences within pathname expansion and pattern matching.

---

Consider the following:

$ touch a A b B c C x X y Y z Z
$ ls
a  A  b  B  c  C  x  X  y  Y  z  Z
$ echo [a-z] # Note the missing uppercase "Z"
a A b B c C x X y Y z
$ echo [A-Z] # Note the missing lowercase "a"
A b B c C x X y Y z Z

Notice when the command echo [a-z] is called, the expected output would be all files with lower case characters. Also, with echo [A-Z], files with uppercase characters would be expected.

---

Standard collations with locales such as en_US have the following order:

aAbBcC...xXyYzZ

• Between a and z (in [a-z]) are ALL uppercase letters, except for Z.
• Between A and Z (in [A-Z]) are ALL lowercase letters, except for a.

See:

     aAbBcC[...]xXyYzZ
     |              |
from a      to      z

     aAbBcC[...]xXyYzZ
      |              |
from  A     to       Z

If you change the LC_COLLATE variable to C it looks as expected:

$ export LC_COLLATE=C
$ echo [a-z]
a b c x y z
$ echo [A-Z]
A B C X Y Z

So, it's not a bug, it's a collation issue.

---

Instead of range expressions you can use POSIX defined character classes, such as upper or lower. They work also with different LC_COLLATE configurations and even with accented characters:

$ echo [[:lower:]]
a b c x y z à è é
$ echo [[:upper:]]
A B C X Y Z

Answer 2

[A-Z] in bash matches all collating elements (characters but call also be sequence of characters like Dsz in Hungarian locales) that sort after A and sort before Z. In your locale, c probably sorts in-between B and C.

$ printf '%s\n' A a á b B c C Ç z Z Ẑ | sort
a
A
á
b
B
c
C
Ç
z
Z
Ẑ

So c or z would be matched by [A-Z], but not Ẑ or a.

$ printf '%s\n' A a á b B c C Ç z Z Ẑ |
pipe>  bash -c 'while IFS= read -r x; do case $x in [A-Z]) echo "$x"; esac; done'
A
á
b
B
c
C
Ç
z
Z

In the C locale, the order would be:

$ printf '%s\n' A a á b B c C Ç z Z Ẑ | LC_COLLATE=C sort
A
B
C
Z
a
b
c
z
Ç
á
Ẑ

So [A-Z] would match A, B, C, Z, but not Ç and still not Ẑ.

If you want to match on upper-case letters (in any script), you can use [[:upper:]] instead. There's no builtin way in bash to only match uppercase letters in the latin script (except by listing them individually).

If you want to match the A to Z English letters without diacritics, you can either use [A-Z] or [[:upper:]] but in the C locale (assuming the data is not encoded in character sets like BIG5 or GB18030 which has several characters whose encoding contains the encoding of those letters) or list them individually ([ABCDEFGHIJKLMNOPQRSTUVWXYZ]).

Note that there is some variation between shells.

For zsh, bash -O globasciiranges (strangely named option introduced in bash-4.3), schily-sh and yash, [A-Z] matches on the characters whose code point is between that of A and that of Z, so would be equivalent to the behaviour of bash in the C locale.

For ash, mksh and ancient shells, same as zsh above but limited to single-byte charsets. That is, in a UTF-8 locale for instance, [É-Ź] would not match on Ó, but since that's [<c3><89>-<c5><b9>], that would match on byte values 0x89 to 0xc5!

ksh93 behaves like bash except that it treats as special cases ranges whose ends both start with lowercase letters or uppercase letters. In that case, it only matches on collating elements that sort between those ends, but that are (or their first character for multi-character collating elements) also lowercase (or uppercase respectively). So [A-Z] there would match on É, but not on e as e does sort between A and Z but is not uppercase like A and Z.

For fnmatch() patterns (as in find -name '[A-Z]') or system regular expressions (as in grep '[A-Z]'), it depends on the system and locale. For instance, on a GNU system here, [A-Z] doesn't match on x in the en_GB.UTF-8 locale, but it does in the th_TH.UTF-8 one. It's unclear to me what information it uses to determine that, but it is apparently based on a lookup table derived from LC_COLLATE locale data).

All behaviours are allowed by POSIX as POSIX leaves the behaviour of ranges unspecified in locales other than the C locale. Now we can argue over the benefits of each approach.

bash's approach makes a lot of sense as with [C-G], we want the characters in between C and G. And using the user's sort order for what determines what's in-between is the most logical approach.

Now, the problem is that it breaks the expectations of a lot of people, especially those people used to the traditional behaviour of pre-Unicode, even pre-internationalisation days. While from a normal user, it makes may sense that [C-I] includes h as the h letter is between C and I and that [A-g] does not include Z, it's a different matter for people having dealt with ASCII only for decades.

That bash behaviour is also different from the [A-Z] range matching in other GNU tools like in GNU regular expressions (as in grep/sed...) or fnmatch() as in find -name.

It also means that what [A-Z] matches varies with the environment, with the OS and with the version of the OS. The fact that [A-Z] matches Á but not Ź is also suboptimal.

For zsh/yash, we use a different sorting order. Instead of relying on the user's notion of character order, we use the character point code values. That has the benefit of being easy to understand, but from a practical point of few, outside of ASCII, it is not very useful. [A-Z] matches the 26 US-english upper-case letters, [0-9] matches decimal digits. There are code points in Unicode that follow the order of some alphabets but that's not generalised and can't be generalised as anyway different people using a same script do not necessarily agree on the order of letters.

For traditional shells and mksh, dash, it's broken (now that most people use multi-byte characters), but primarily because they don't have multi-byte support yet. Adding multi-byte support to shells like bash and zsh has been a huge effort and is still ongoing. yash (a Japanese shell) was initially designed with multi-byte support from the start.

ksh93's approach has the benefit to be consistent with the system's regular expressions or fnmatch() (or at least appears to at least on GNU systems). There, it doesn't break some people's expectation as [A-Z] doesn't include lower case letters, [A-Z] includes É (and Á, but not Ź). It's not consistent with sort or generally strcoll() order.

Answer 3

It's intended and documented in bash documentation, pattern matching section. The range expression [X-Y] will be included any characters between X and Y using the current locale’s collating sequence and character set:

LC_ALL=en_US.utf8 bash -c 'case b in [A-Z]) echo yes; esac' 
yes

You can see, b sorted between A and Z in en_US.utf8 locale.

You have some choices to prevent this behavior:

# Setting LC_ALL or LC_COLLATE to C
LC_ALL=C bash -c 'echo [A-Z]*'

# Or using POSIX character class
LC_ALL=C bash -c 'echo [[:upper:]]*'

or enable globasciiranges (with bash 4.3 and above):

bash -O globasciiranges -c 'echo [A-Z]*'

Answer 4

I observed this behavior on a new Amazon EC2 instance. Since the OP didn't offer an MCVE, I'll post one:

$ cd $(mktemp -d)
$ touch foo
$ echo [A-Z]*     # prepare for a surprise!
foo

$ echo $BASH_VERSION
4.1.2(1)-release
$ uname -a
Linux spinup-tmp12 3.14.27-25.47.amzn1.x86_64 #1 SMP Wed Dec 17 18:36:15 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

$ env | grep LC_  # no locale, let's set one
$ LC_ALL=C
$ echo [A-Z]*
[A-Z]*

$ unset LC_ALL    # ok, good. what if we go back to no locale?
$ echo [A-Z]*
foo

So, not having my LC_* set leads bash 4.1.2(1)-release on Linux to produce apparently odd behavior. I can reliably toggle the odd behavior by setting and unsetting the respective locale variables. Unsurprisingly, this behavior appears consistent through exporting:

$ export LC_ALL=C
$ bash
$ echo [A-Z]*
[A-Z]*
$ exit
$ echo $SHLVL
1
$ unset LC_ALL
$ bash
$ echo [A-Z]*
foo

While I'm seeing bash behave as Stéphane "Shellshock" Chazelas answered, I think the bash documentation on pattern matching is buggy:

For example, in the default C locale, ‘[a-dx-z]’ is equivalent to ‘[abcdxyz]’

I read that sentence (emphasis mine) as "if the relevant locale variables are not set, then bash will default to the C locale". Bash does not appear to be doing that. Instead it appears to be defaulting to a locale where the characters are sorted in dictionary order with diacritic folding:

$ echo [A-E]*
[A-E]*
$ echo [A-F]*
foo
$ touch "évocateur"
$ echo [A-F]*
foo évocateur

I think it'd be good for bash to document how it will behave when LC_* (specifically LC_CTYPE and LC_COLLATE) are undefined. But in the mean time, I'll share some wisdom:

... you have to be very careful with [character ranges] because they will not produce the expected results unless properly configured. For now, you should avoid using them and use character classes instead.

and

If you're really proper, and/or are scripting for a multi-locale environment, it's probably best to make sure you know what your locale variables are when you're matching files, or to be sure that you're coding in a completely generic way.

---

Update Based on @G-Man comment, let's look deeper into what's happening:

$ env | grep LANG
LANG=en_US.UTF-8

Ah, ha! That explains the collation seen earlier. Let's remove all the locale variables:

$ unset LANG LANGUAGE LC_ALL
$ env | grep 'LC_|LANG'
$ echo [A-Z]*
[A-Z]*

There we go. Now bash operates consistently with respect to documentation on this Linux system. If any of the locale variables are set (LANGUAGE, LANG, LC_COLLATE, LC_CTYPE, LC_ALL, etc.) then Bash uses those according to its manual. Otherwise, bash falls back to C.

The Wooledge bash FAQ has this to say:

On recent GNU systems, the variables are used in this order. If LANGUAGE is set, use that, unless LANG is set to C, in which case LANGUAGE is ignored. Also, some programs simply don't use LANGUAGE at all. Otherwise, if LC_ALL is set, use that. Otherwise, if the specific LC_* variable that covers this usage is set, use that. (For example, LC_MESSAGES covers error messages.) Otherwise, use LANG.

So the apparent problem, both in operation and documentation, can be explained by looking at the total sum of all locale driving variables.

Answer 5

Locale can change what characters are matched by [A-Z]. Use

(LC_ALL=C; rm [A-Z]*)

to eliminate the influence. (I used a subshell to localize the change).

Answer 6

As has been already said, this is a "collating order" issue.

The range a-z may contain upper case letters in some locales:

     aAbBcC[...]xXyYzZ
     |              |
from a      to      z

The correct solution since bash 4.3 is to set the option globasciiranges:

shopt -s globasciiranges

to make bash act as if LC_COLLATE=C has been set in globing ranges.

Answer 7

It seems that I found the right answer to my own question:

Bash is buggy as it does not manage it's own locale. So setting LC_* in a bash process is without effect in that shell process.

If you set the LC_COLLATE=C and then start another bash, the globbing works as expected in the new bash process.