Why should 'Character Classes' be preferred over 'Character Ranges' In Shell (Bash)?

Question

The Linux Command Line (Book - page count 47) says:

... you have to be very careful with them [character ranges] because they will not produce the expected results unless properly configured. For now, you should avoid using them and use character classes instead.

The book gives no reason, other than that.

Question - 1: So, why exactly should Character Classes (e.g. [:alnum:], [:alpha:], [:digit:], etc) be preferred over Character Ranges (e.g. [a-z], [A-Z], [0-9], etc)?

Question - 2: Does [:alpha:] stand for [a-z], [A-Z], and upper and lower-case alphabets from other languages too? And similarly, does [:digit:] include numerals from other languages too? If they match, that is.

(Two questions, I know, but in this case, they are pretty much interrelated, IMO.)

Answer 1

According to the bash manpage, the LC_COLLATE environment variable affects character ranges, exactly as per Hauke Laging's answer:

LC_COLLATE This variable determines the collation order used when sorting the results of pathname expansion, and determines the behavior of range expressions, equivalence classes, and collating sequences within pathname expansion and pattern matching.

On the other hand, LC_CTYPE affects character classes:

LC_CTYPE This variable determines the interpretation of characters and the behavior of character classes within pathname expansion and pattern matching.

What this means is that both cases are potentially problematic if you're thinking in a English, left-to-right, Latin alphabet, Arabic-digit context.

If you're really proper, and/or are scripting for a multi-locale environment, it's probably best to make sure you know what your locale variables are when you're matching files, or to be sure that you're coding in a completely generic way.

It's very difficult to foresee some situations though, unless you've studied linguistics.

However, I don't know of a Latin-using locale that changes the order of letters, so [a-z] would work. There are extensions to the Latin alphabet that collate ligatures and diacriticals differently. However, here's a little experiment:

mkdir /tmp/test
cd /tmp/test
export LC_CTYPE=de_DE.UTF-8
export LC_COLLATE=de_DE.UTF-8
touch Grüßen
ls G* # This says ‘Grüßen’
ls *[a-z]en # This says nothing!
ls *[a-zß]en # This says ‘Grüßen’
ls Gr[a-z]*en # This says nothing!

This is interesting: at least for German, neither diacriticals like ü nor ligatures like ß are folded into latin characters. (either that, or I messed up the locale change!)

This may be bad for you, of course, if you're trying to find filenames that start with a letter, use [a-z]* and apply it to a file that starts with ‘Ä’.

Answer 2

At least when using bash 4.2 on OS X, the UTF-8 locales seem to use the ASCII collation order, but the ISO 8859-1 locales don't in some contexts:

$ LC_ALL=en_US.UTF-8 tr A-C 1-9 <<< B
2
$ LC_ALL=en_US.ISO8859-1 tr A-C 1-9 <<< B
6
$ LC_ALL=en_US.UTF-8 grep [A-Z] <<< ä
$ LC_ALL=en_US.ISO8859-1 grep [A-Z] <<< ä
ä

In some environments the UTF-8 locales also use different collation orders.

[:upper:] and [:lower:] also include non-ASCII characters in many locales. If you only want to match ASCII characters, use something like this:

LC_ALL=C tr a-zA-Z n-za-mN-ZA-M

If LC_ALL has been set to something else, LC_COLLATE=C or LANG=C wouldn't work.

Answer 3

"Other languages", that's it. Different locales may have different sort orders. So in theory it may be that a-z is not the same with another locale. Ranges become difficult it you want to match everything. What is the first, what the last char?

The openSUSE people are so paranoid about that when checking usernames / passwords that they do it this way: [abcdefghi...]

I never thought of numerals in other languages / character sets. Interesting point.