Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

The Linux Command Line (Book - page count 47) says:

... you have to be very careful with them [character ranges] because they will not produce the expected results unless properly configured. For now, you should avoid using them and use character classes instead.

The book gives no reason, other than that.

Question - 1: So, why exactly should Character Classes (e.g. [:alnum:], [:alpha:], [:digit:], etc) be preferred over Character Ranges (e.g. [a-z], [A-Z], [0-9], etc)?

Question - 2: Does [:alpha:] stand for [a-z], [A-Z], and upper and lower-case alphabets from other languages too? And similarly, does [:digit:] include numerals from other languages too? If they match, that is.

(Two questions, I know, but in this case, they are pretty much interrelated, IMO.)

share|improve this question
    
Strongly related: Does (should) LC_COLLATE affect character ranges? –  Gilles Apr 17 '13 at 23:39

3 Answers 3

up vote 2 down vote accepted

According to the bash manpage, the LC_COLLATE environment variable affects character ranges, exactly as per Hauke Laging's answer:

LC_COLLATE This variable determines the collation order used when sorting the results of pathname expansion, and determines the behavior of range expressions, equivalence classes, and collating sequences within pathname expansion and pattern matching.

On the other hand, LC_CTYPE affects character classes:

LC_CTYPE This variable determines the interpretation of characters and the behavior of character classes within pathname expansion and pattern matching.

What this means is that both cases are potentially problematic if you're thinking in a English, left-to-right, Latin alphabet, Arabic-digit context.

If you're really proper, and/or are scripting for a multi-locale environment, it's probably best to make sure you know what your locale variables are when you're matching files, or to be sure that you're coding in a completely generic way.

It's very difficult to foresee some situations though, unless you've studied linguistics.

However, I don't know of a Latin-using locale that changes the order of letters, so [a-z] would work. There are extensions to the Latin alphabet that collate ligatures and diacriticals differently. However, here's a little experiment:

<!-- language: lang-bash -->
mkdir /tmp/test
cd /tmp/test
export LC_CTYPE=de_DE.UTF-8
export LC_COLLATE=de_DE.UTF-8
touch Grüßen
ls G* # This says ‘Grüßen’
ls *[a-z]en # This says nothing!
ls *[a-zß]en # This says ‘Grüßen’
ls Gr[a-z]*en # This says nothing!

This is interesting: at least for German, neither diacriticals like ü nor ligatures like ß are folded into latin characters. (either that, or I messed up the locale change!)

This may be bad for you, of course, if you're trying to find filenames that start with a letter, use [a-z]* and apply it to a file that starts with ‘Ä’.

share|improve this answer
    
Even after reading the two answers, one thing isn't clear: Why Character Classes over Character Ranges? Why the preference, precisely? –  its_me Apr 17 '13 at 13:35
1  
As Alexios implies (+1), I'd think locale cases where [a-z] turns to include more or less than a-z are going to be very unusual -- the issue is that a-z is not all alphabetic characters in many locales. So if you are looking for words, the [:alpha:] character class provides a big advantage: it is portable across locales, which [a-z] obviously is not. –  goldilocks Apr 17 '13 at 13:56
    
@Alexios: if somewhere along the line you are not using de_DE.UTF-8 (process, filesystem, etc.) but something like de_DE.ISO-8859-1 you will get your odd results. I am using the ISO encoding and I get Grüßen when ls Gr[a-z]*en –  Bananguin Apr 17 '13 at 14:38
1  
@TheoneManis because they may behave in unexpected ways if you're using Unicode. More unexpected than character ranges, that is. Unless you're using character ranges with Unicode codepoints greater than U+007F, I suppose. It may be down to the author's personal preference (the perceived lesser of two evils by personal experience). –  Alexios Apr 18 '13 at 19:45

"Other languages", that's it. Different locales may have different sort orders. So in theory it may be that a-z is not the same with another locale. Ranges become difficult it you want to match everything. What is the first, what the last char?

The openSUSE people are so paranoid about that when checking usernames / passwords that they do it this way: [abcdefghi...]

I never thought of numerals in other languages / character sets. Interesting point.

share|improve this answer
    
Even after reading the two answers, I am not sure about this -- Why Character Classes over Character Ranges? Why the preference, precisely? –  its_me Apr 17 '13 at 13:36
    
I must admit that I didn't know that character classes are affected, too, by the locale. And if [:digit:] can be something you have never seen before then it probably makes sense to prefer [0-9]. Maybe these rules depend on what exactly one is doing. –  Hauke Laging Apr 17 '13 at 13:53

At least when using bash 4.2 on OS X, the UTF-8 locales seem to use the ASCII collation order, but the ISO 8859-1 locales don't in some contexts:

$ LC_ALL=en_US.UTF-8 tr A-C 1-9 <<< B
2
$ LC_ALL=en_US.ISO8859-1 tr A-C 1-9 <<< B
6
$ LC_ALL=en_US.UTF-8 grep [A-Z] <<< ä
$ LC_ALL=en_US.ISO8859-1 grep [A-Z] <<< ä
ä

In some environments the UTF-8 locales also use different collation orders.

[:upper:] and [:lower:] also include non-ASCII characters in many locales. If you only want to match ASCII characters, use something like this:

LC_ALL=C tr a-zA-Z n-za-mN-ZA-M

If LC_ALL has been set to something else, LC_COLLATE=C or LANG=C wouldn't work.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.