Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Collation order through LC_COLLATE defines not only the sort order of individual characters, but also the meaning of character ranges. Or does it? Consider the following snippet:

unset LANGUAGE LC_ALL
echo B | LC_COLLATE=en_US grep '[a-z]'

Intuitively, B isn't in [a-z], so this shouldn't output anything. That's what happens on Ubuntu 8.04 or 10.04. But on some machines running Debian lenny or squeeze, B is found, because the range a-z includes everything that's between a and z in the collation order, including the capital letters B through Z.

All systems tested do have the en_US locale generated. I also tried varying the locale: on the machines where B is matched above, the same happens in every available locale (mostly latin-based: {en_{AU,CA,GB,IE,US},fr_FR,it_IT,es_ES,de_DE}{iso8859-1,iso8859-15,utf-8}, also Chinese locales) except Japanese (in any available encoding) and C/POSIX.

What do character ranges mean in regular expressions, when you go beyond ASCII? Why is there a difference between some Debian installations on the one hand, and other Debian installations and Ubuntu on the other? How do other systems behave? Who's right, and who should have a bug reported against?

(Note that I'm specifically asking about the behavior of character ranges such as [a-z] in en_US locales, primarily on GNU libc-based systems. I'm not asking how to match lowercase letters or ASCII lowercase letters.)


On two Debian machines, one where B is in [a-z] and one where it isn't, the output of LC_COLLATE=en_US locale -k LC_COLLATE is

collate-nrules=4
collate-rulesets=""
collate-symb-hash-sizemb=1
collate-codeset="ISO-8859-1"

and the output of LC_COLLATE=en_US.utf8 locale -k LC_COLLATE is

collate-nrules=4
collate-rulesets=""
collate-symb-hash-sizemb=2039
collate-codeset="UTF-8"
share|improve this question
    
The results on my systems (PLD-Linux) are frankly disconcerting. It seems most non-english collations include capital variants in character ranges, but at least for me en_US does not. The English collation set, meanwhile, includes SOME characters from other alphabets but not a very complete set, so [a-z] includes much more than 26 characters, but not the full Turkish alphabet for example. –  Caleb Jul 2 '11 at 11:52
1  
Doesn't reproduce on a Debian Lenny instance I've had handy. Didn't check if en_US is generated, though. –  alex Jul 2 '11 at 13:35
1  
@alex If the locale isn't generated, the C locale is used as a fallback, and its collation order is straight byte values, so B won't be matched. Test in a locale that appears in the output of locale -a. –  Gilles Jul 2 '11 at 13:43
1  
Note that en_US is NOT the same as en_US.utf8, and typically means en_US.iso-8859-1, depending on exactly what you have installed. If en_US (with no suffix) doesn't appear in the output of locale -a you don't actually have this locale. What does LC_COLLATE=en_US locale -k LC_COLLATE show? –  Neil Mayhew Jul 4 '11 at 13:26
1  
This has since turned up in a practical rather than theoretical question here: Why are capital letters included in a range of lower-case letters in an awk regex? –  Caleb Aug 24 '11 at 21:22
show 5 more comments

3 Answers 3

If you are using anything other than the C locale, you shouldn't be using ranges like [a-z] since these are locale-dependent and don't always give the results you would expect. As well as the case issue you've already encountered, some locales treat characters with diacritics (eg á) the same as the base character (ie a).

Instead, use a named character class:

echo B | grep '[[:lower:]]'

This will always give the correct result for the locale. However, you need to choose the locale to reflect the meaning of both your input text and the test you are trying to apply.

share|improve this answer
    
I know that, it isn't what I asked. I'm specifically asking about what an explicit range means, and why different distributions (even with GNU libc and GNU grep) have different behaviors. (Downvoted because even though what you say is correct, it's irrelevant.) –  Gilles Jul 3 '11 at 22:47
1  
My point is that the meaning of an explicit range is locale-dependent, and different systems are not required to define their locales the same way, so this is not a bug. Technically, you are abusing the system, so you shouldn't be surprised at getting "undefined" behaviour. Also, several people have commented that they can't reproduce the behaviour on their Debian systems, so there seems to be something unusual about your system(s). –  Neil Mayhew Jul 4 '11 at 13:17
    
I know that the behavior of ranges depends on the locales. I'm asking how, and surprised that different systems using Glibc (and, it turns out, even different installations of the same Debian release) have different behaviors. I've added the output of locale -k to my question; it's identical on two Debian machines, one where B is in the range and one where it isn't. BTW I'm not root on either machine (so it's not something peculiar that I do as an admin). –  Gilles Jul 4 '11 at 19:46
add comment

Ranges in regular expressions should observe the collation setting. Here is the relevant standard: http://pubs.opengroup.org/onlinepubs/007908799/xbd/re.html (look for "range expressions"). So echo B | LC_COLLATE=en_US grep '[a-z]' should output B given a sensible definition of the respective locale. I can't explain why this sometimes doesn't work for you, but I would be very surprised if I encountered this on a non-ancient system that is properly installed and configured.

share|improve this answer
1  
echo B | LC_COLLATE=en_US.utf8 grep '[a-z]' Doesn't print anything on Ubuntu 12.04 with grep 2.10. Doesn't print anything on Centos 6.5 with grep 2.6.3. Does work on Debian 6.0.8 with grep 2.6.3. –  IDAllen Jan 27 at 6:20
add comment

ASCII characters are fixed for any locale. I hope Debian lenny or squeeze have bug in locale sources.

you can see mklocale to understand how locale files are generated.

share|improve this answer
    
Um, all POSIX locales are ASCII-based, but there are unix systems out there that support EBCDIC encodings. I don't see how that's relevant here: the question involves collation order, which in most locales is not the order of the characters in the encoding. –  Gilles Nov 4 '11 at 15:55
    
characters for en_US have fixed order even if locale is UTF-8 or UTF-16 –  Eir Nym Nov 4 '11 at 16:59
    
Oh, you mean ASCII characters are fixed for any encoding. That does match my observations, and it makes sense that they would be, but I don't know that it would be the case everywhere, and in any case I don't see how this relates to the question. –  Gilles Nov 4 '11 at 17:29
    
ASCII encoding is standard since 1963 and it must be fixed. If some symbols from it is broken, you'll have bug. In another hand, grep(1) may have -i defined in environment(check all of GREP_OPTIONS,LC_ALL, LC_COLLATE, LANG,LC_MESSAGES,LC_CTYPE,POSIXLY_CORRECT to be sure) and you'll get big letter is in lower characters range. –  Eir Nym Nov 4 '11 at 17:55
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.