3

In this question there is a comment which says:

All of this from not understanding what "in the POSIX locale" means. (-: You should really try matching Greek lowercase letters with (say) sed and [[:lower:]] and a handy el_GR.UTF-8 locale set. – JdeBP

So: What does "in the POSIX locale" mean?

Bonus: is this valid in POSIX in locales other than C ?

LC_ALL=C grep '[[:lower:]]' file
0

2 Answers 2

4

When POSIX says (for “upper”)

In the POSIX locale, only:

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

shall be included:

it’s defining the POSIX locale. It means that in the context of the POSIX locale, the “upper” character class consists only of the listed characters.

POSIX doesn’t define any other locale, but systems are free to do so. As a result, you can’t rely on grep '[[:lower:]]' file doing anything useful in a locale other than POSIX, if you limit yourself to POSIX only. However, many systems do define other locales and their character classes. For example, on a system using the GNU C library’s locales,

$ echo 'α' | LC_ALL=el_GR.UTF-8 grep '[[:lower:]]'
α

(This assumes the el_GR.UTF-8 locale is available; on Debian derivatives, a quick way of ensuring that is to install the locales-all package.)

5
  • Perfect, @StephenKitt, you have nailed the core issue in the head. And yes, all you have written is correct. But the question remains: Is LC_ALL=el_GR.UTF-8 grep '[[:lower:]]' valid according to the POSIX spec (I don't mean the POSIX locale, or according to some other (non-POSIX) implementation (glibc)). I suspect it is not valid, correct?
    – user232326
    Mar 8, 2020 at 22:42
  • 2
    POSIX doesn’t specify other locales, which means that you can’t rely on this el_GR behaviour on a purely POSIX system, but POSIX doesn’t forbid it, so it’s not invalid. Mar 8, 2020 at 22:51
  • Well, yes, my bad: I suspect the word is: Undefined (instead of invalid), yes, sorry. So, the command is Undefined and the output of such command could be anything, even "format your disk" (as a joke), correct?
    – user232326
    Mar 8, 2020 at 22:57
  • No need to apologise ;-). “Undefined” is indeed more appropriate. However that doesn’t imply C-style undefined behaviour; what’s undefined is character class contents in specific locales, not character classes as a whole. A character class still defines characters (only). POSIX does define base requirements for some character classes (“automatically included characters”); for example, a locale’s definition of “upper” should include characters A to Z from the POSIX definition. But POSIX is careful to not even require that; extensions are allowed to prevent the automatic inclusion. Mar 9, 2020 at 8:56
  • Put another way, what’s undefined (in POSIX) isn’t the grep command, it’s the set of characters matched by [[:lower:]] when using the el_GR.UTF-8 LC_CTYPE; so the extent of the “undefinition” is the matching behaviour of the grep command, that’s all. grep could match more or less than expected of the input text; but it can’t invent new input, or new behaviours. Mar 9, 2020 at 8:58
0

I haven't fully understood your suggestion (in the other question). However it does appear you have misinterpreted the use case of the POSIX locale.

Your suggestion seems to be that the locale is not very user friendly. And that it should be adjusted to make it so. However user friendly isn't the point of the posix local, that's the job of other locales with other rules.

The job of the posix local is to be predictable. Indeed it also benefits from being simple. Once you start including other languages into a standard it is very difficult to stop, and the result is a standard that needs a lot of maintenance and is hard to implement.

There is no way that I, as a developer know every rule about every language, how to sort them in "alphabetical" order and indeed know which are upper or lower case. The rules for non-english languages can be very complex indeed. So if you adjust the Posix local to include the rules for every other language, you would make it's behaviour very unpredictable indeed.

Changes to the posix locale could indeed be crippeling to software where older systems did not match the behaviour of newer systems. (See anecdote below)


Anecdote

One of the most obscure and difficult to diagnose bugs I've ever seen was in an ETL tool "randomly" dropping rows. After painstaking analysis, it transpired that the software worked fine, but only if run with LC_COLLATE=C. This was because, when matching records, it relied on the sort order from a database matching its own internal sort order. The developers had simply never noticed that in some locales numbers would sort text alphabetically "0, -1, 1, -2, 2" instead of "-1, -2, 1, 2".

Quite aside from user interaction, what this shows is a genuine need for systems to behave in a predictable way that can be common to all systems.

3
  • I am for a "predictable way", indeed. A [[:as:lower:]] (ascii-lower) could have no misinterpretations (IMhO), the range is very well defined and without problems for developers. What leads to problems is that the general [0-9] (for example) has been shifting in meaning as time pass. I believe that something should/could be done about that, and ranges in general (for other locales).
    – user232326
    Mar 8, 2020 at 22:19
  • Not true. It's easy to forget UTF-8 isnt ubiquitous. As far as I know, other locales deliberately state that they are "UTF-8" eg en_gb.utf-8. I don't believe posix locale does this. UTF-8 was designed to share the first 127 byte values with ascii making it very compatible with ascii. Posix locale, I believe, shares this deliberate compatibility. To add logic based on unicode character codes above 127 will break this rule. If I fed in a file encoded with windows 1252 (a form of extended ascii), I would get very unexpected behaviour indeed. Mar 8, 2020 at 23:49
  • 2
    Be careful. "How to sort" does depend on language. For one, Spanish and German share the "ü" character, but for sorting it counts as a plain "u" in Spanish, and sorts after "u" in German. Spanish got recently cleaned up, but it did hale "ch" and "ll"that counted as one letter (after "c" and "l", respectively), in German (English, French, ...) that isn't so.
    – vonbrand
    Mar 9, 2020 at 16:36

You must log in to answer this question.