Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
In this question there is a comment which says:
All of this from not understanding what "in the POSIX locale" means. (-: You should really try matching Greek lowercase letters with (say) sed and [[:lower:]] and a handy el_GR.UTF-8 locale set. – JdeBP
So: What does "in the POSIX locale" mean?
Bonus: is this valid in POSIX in locales other than C ?
LC_ALL=C grep '[[:lower:]]' file
When POSIX says (for “upper”)
In the POSIX locale, only:
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
shall be included:
it’s defining the POSIX locale. It means that in the context of the POSIX locale, the “upper” character class consists only of the listed characters.
POSIX doesn’t define any other locale, but systems are free to do so. As a result, you can’t rely on grep '[[:lower:]]' file doing anything useful in a locale other than POSIX, if you limit yourself to POSIX only. However, many systems do define other locales and their character classes. For example, on a system using the GNU C library’s locales,
$ echo 'α' | LC_ALL=el_GR.UTF-8 grep '[[:lower:]]'
α
(This assumes the el_GR.UTF-8 locale is available; on Debian derivatives, a quick way of ensuring that is to install the locales-all package.)
LC_ALL=el_GR.UTF-8 grep '[[:lower:]]' valid according to the POSIX spec (I don't mean the POSIX locale, or according to some other (non-POSIX) implementation (glibc)). I suspect it is not valid, correct?
el_GR behaviour on a purely POSIX system, but POSIX doesn’t forbid it, so it’s not invalid.
Mar 8, 2020 at 22:51
grep command, it’s the set of characters matched by [[:lower:]] when using the el_GR.UTF-8 LC_CTYPE; so the extent of the “undefinition” is the matching behaviour of the grep command, that’s all. grep could match more or less than expected of the input text; but it can’t invent new input, or new behaviours.
Mar 9, 2020 at 8:58
I haven't fully understood your suggestion (in the other question). However it does appear you have misinterpreted the use case of the POSIX locale.
Your suggestion seems to be that the locale is not very user friendly. And that it should be adjusted to make it so. However user friendly isn't the point of the posix local, that's the job of other locales with other rules.
The job of the posix local is to be predictable. Indeed it also benefits from being simple. Once you start including other languages into a standard it is very difficult to stop, and the result is a standard that needs a lot of maintenance and is hard to implement.
There is no way that I, as a developer know every rule about every language, how to sort them in "alphabetical" order and indeed know which are upper or lower case. The rules for non-english languages can be very complex indeed. So if you adjust the Posix local to include the rules for every other language, you would make it's behaviour very unpredictable indeed.
Changes to the posix locale could indeed be crippeling to software where older systems did not match the behaviour of newer systems. (See anecdote below)
Anecdote
One of the most obscure and difficult to diagnose bugs I've ever seen was in an ETL tool "randomly" dropping rows. After painstaking analysis, it transpired that the software worked fine, but only if run with LC_COLLATE=C. This was because, when matching records, it relied on the sort order from a database matching its own internal sort order. The developers had simply never noticed that in some locales numbers would sort text alphabetically "0, -1, 1, -2, 2" instead of "-1, -2, 1, 2".
Quite aside from user interaction, what this shows is a genuine need for systems to behave in a predictable way that can be common to all systems.
[[:as:lower:]] (ascii-lower) could have no misinterpretations (IMhO), the range is very well defined and without problems for developers. What leads to problems is that the general [0-9] (for example) has been shifting in meaning as time pass. I believe that something should/could be done about that, and ranges in general (for other locales).
en_gb.utf-8. I don't believe posix locale does this. UTF-8 was designed to share the first 127 byte values with ascii making it very compatible with ascii. Posix locale, I believe, shares this deliberate compatibility. To add logic based on unicode character codes above 127 will break this rule. If I fed in a file encoded with windows 1252 (a form of extended ascii), I would get very unexpected behaviour indeed.
Mar 8, 2020 at 23:49