0

Having a file with all the (printable) ascii characters:

$ printf '%b' "$(printf '\\U%x\n' {32..126})" > file

That could be sorted (using tr to reduce the long output to one line):

$ sort file | tr -d '\n'
 !"#%&'()*+,-./:;<=>?@[\]^_`{|}~$0123456789aAbBcCdDeEfFgGhHiIjJkKlLmMnNoOpPqQrRsStTuUvVwWxXyYzZ

Shows that on a Debian buster computer using a en_US.utf8 locale the collate sorting (for single characters) has all punctuation first, then numbers and then intermixed case letters. That is: aAbB, lower and upper case letters together.

Let's assume that that is correct, and is what the user (Me) wants to happen for collating.

However, on the same system, without any other change, this happens:

$ grep '[a-z]' file | tr -d '\n'
abcdefghijklmnopqrstuvwxyz

That is the range a-z is translated by something to only lowercase ASCII letters.

Who is doing the translation and how that could be controlled or changed ?

I am not asking about what lowercase letters are nor what a [a-z] should mean or someone want that to mean.

I expect that [a-z] is a range that starts on a and ends on z in the collating order.

I understand that some other users want that a [a-z] mean the same as "lowercase" in any locale. And I might "live with that" by default.

But how could I control and/or change that, if needed be ? Where is the knob to change that ?

No, changing the collate files doesn't help, something is above and beyond that and enforces the personal view that a [a-z] must mean ASCII lowercase all the time in all locales.

  • "I expect that [a-z] is a range that starts on a and ends on z in the collating order." – Note this is exactly what happens. It seems you expect "in the collating sequence". I think "collating order" and "collating sequence" are different and this fact plays a major role here. – Kamil Maciorowski Apr 5 at 13:41
3

I read what POSIX states. My interpretation is there are two non-equivalent concepts:

  • collation sequence (collating sequence)
  • collation order

Relevant fragments [emphasis mine]:

The LC_COLLATE category provides a collation sequence definition for numerous utilities in the Shell and Utilities volume of POSIX.1-2017 (ls, sort, and so on), regular expression matching (see Regular Expressions), and the strcoll(), strxfrm(), wcscoll(), and wcsxfrm() functions in the System Interfaces volume of POSIX.1-2017.

A collation sequence definition shall define the relative order between collating elements (characters and multi-character collating elements) in the locale. This order is expressed in terms of collation values; that is, by assigning each element one or more collation values (also known as collation weights). […]

The order_start keyword shall precede collation order entries and also define the number of weights for this collation sequence definition and other collation rules.

The collation order as defined in this section affects the interpretation of bracket expressions in regular expressions (see RE Bracket Expression).

For sort the collation sequence matters, i.e. the weights. For grep '[a-z]' the collation order matters, i.e. the order of collation order entries.

Unfortunately only the collation sequence is defined explicitly, so there is no clear indication the collation order is a different concept.

Collation Sequence
The relative order of collating elements as determined by the setting of the LC_COLLATE category in the current locale. The collation sequence is used for sorting and is determined from the collating weights assigned to each collating element. In the absence of weights, the collation sequence is the order in which collating elements are specified between order_start and order_end keywords in the LC_COLLATE category.


In my Debian 9, when it comes to LC_COLLATE, many locales eventually refer to iso14651_t1_common (i.e. /usr/share/i18n/locales/iso14651_t1_common). The relevant fragment of the file looks like this:

<U0061> <a>;<BAS>;<MIN>;IGNORE # 198 a
<U00AA> <a>;<PCL>;<EMI>;IGNORE # 199 ª
<U00E1> <a>;<ACA>;<MIN>;IGNORE # 200 á
[…]
<U0062> <b>;<BAS>;<MIN>;IGNORE # 233 b
<U0253> <b>;<CRL>;<MIN>;IGNORE # 234 ɓ
<U1E03> <b>;<PCT>;<MIN>;IGNORE # 235 ḃ
[…]
<U007A> <z>;<BAS>;<MIN>;IGNORE # 507 z
<U017A> <z>;<ACA>;<MIN>;IGNORE # 508 <z'>
<U017E> <z>;<CAR>;<MIN>;IGNORE # 509 <z<>
[…]
<U0041> <a>;<BAS>;<CAP>;IGNORE # 517 A
<U00C1> <a>;<ACA>;<CAP>;IGNORE # 518 Á
<U00C0> <a>;<GRA>;<CAP>;IGNORE # 519 À
[…]
<U0042> <b>;<BAS>;<CAP>;IGNORE # 550 B
<U1E02> <b>;<PCT>;<CAP>;IGNORE # 551 <B.>
<U1E04> <b>;<BPT>;<CAP>;IGNORE # 552 Ḅ
[…]
<U005A> <z>;<BAS>;<CAP>;IGNORE # 813 Z
<U0179> <z>;<ACA>;<CAP>;IGNORE # 814 <Z'>
<U017D> <z>;<CAR>;<CAP>;IGNORE # 815 <Z<>

This is the collation order. [a-z] does not contain A because the entry for A (<U0041>) is not between the entries for a and z.

Still the entries for a and A specify the same collating symbol <a>. Similarly b and B specify <b>. This translates to weights:

Weights shall be expressed as characters (in any of the forms specified in Locale Definition), <collating-symbol>s, <collating-element>s, an ellipsis, or the special symbol IGNORE. A single character, a <collating-symbol>, or a <collating-element> shall represent the relative position in the character collating sequence of the character or symbol, rather than the character or characters themselves. Thus, rather than assigning absolute values to weights, a particular weight is expressed using the relative order value assigned to a collating element based on its order in the character collation sequence.

In the file <a> and <b> are defined in this order:

collating-symbol <a>
collating-symbol <b>

This makes the relevant subsequence of collation sequence be aAbB. This is what matters to sort.


To confirm this I (temporarily) moved the following collation order entry

<U004B> <k>;<BAS>;<CAP>;IGNORE # 649 K

to a position just before the entry for v, i.e. somewhere in between a and z. I rebuilt my locales with locale-gen. Now sort file | tr -d '\n' still returns …iIjJkKlLmM… (the weights did not change, the collation sequence did not change), but grep '[a-z]' file | tr -d '\n' yields:

Kabcdefghijklmnopqrstuvwxyz

It means I made K belong to [a-z] by changing the collation order.

If you want grep '[a-z]' file | tr -d '\n' to return a permutation of aAbBcCdDeEfFgGhHiIjJkKlLmMnNoOpPqQrRsStTuUvVwWxXyYzZ rather than abcdefghijklmnopqrstuvwxyz, then you need to use a locale with different collation order. It may be a custom locale.

| improve this answer | |
  • Thanks, I have been able to reproduce your changes, I confirm that what you wrote works. I do not understand (yet) what is going on, but I will, As soon as there more news I'll let you know. – Isaac Apr 6 at 5:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.