27

ɛ ("Latin epsilon") is a letter used in certain African languages, usually to represent the vowel sound in English "bed". In Unicode it's encoded as U+025B, very distinct from everyday e.

However, if I sort the following:

eb
ed
ɛa
ɛc

it seems that sort considers ɛ and e equivalent:

ɛa
eb
ɛc
ed

What's going on here? And is there a way to make ɛ and e distinct for sorting purposes?

7
  • 23
    sorting rules are called 'collation', if that helps your googling Oct 26, 2018 at 21:28
  • 1
    Try to put a certain number of ea mixed with ɛa inside a text file and sort it. You will see that it always sorts ea before ɛa. So, no they are not considered equal.
    – Bakuriu
    Oct 27, 2018 at 9:21
  • Might be an obvious point, but I haven't seen it suggested explicitly yet: if you are sorting words in $(certain_african_language), the natural thing to do is setting the locale to $(certain_african_language). Oct 28, 2018 at 9:13
  • @FedericoPoloni A very good point! Unfortunately I haven't been able to find any locale made for this language.
    – Draconis
    Oct 28, 2018 at 15:30
  • 1
    @GermánBouzas This is specifically "Latin epsilon", a form designed to fit in with the Latin alphabet. They look pretty much the same, but Latin epsilon is U+025B, while Greek epsilon is U+03B5.
    – Draconis
    Nov 1, 2018 at 13:22

3 Answers 3

69

No, it doesn't consider them as equivalent, they just have the same primary weight. So that, in first approximation, they sort the same.

If you look at /usr/share/i18n/locales/iso14651_t1_common (as used as basis for most locales) on a GNU system (here with glibc 2.27), you'll see:

<U0065> <e>;<BAS>;<MIN>;IGNORE # 259 e
<U025B> <e>;<PCL>;<MIN>;IGNORE # 287 ɛ
<U0045> <e>;<BAS>;<CAP>;IGNORE # 577 E

e, ɛ and E have the same primary weight, e and E same secondary weight, only the third weight differentiates them.

When comparing strings, sort (the strcoll() standard libc function is uses to compare strings) starts by comparing the primary weights of all characters, and only go for the second weight if the strings are equal with the primary weights (and so on with the other weights).

That's how case seems to be ignored in the sorting order in first approximation. Ab sorts between aa and ac, but Ab can sort before or after ab depending on the language rule (some languages have <MIN> before <CAP> like in British English, some <CAP> before <MIN> like in Estonian).

If e had the same sorting order as ɛ, printf '%s\n' e ɛ | sort -u would return only one line. But as <BAS> sorts before <PCL>, e alone sorts before ɛ. eɛe sorts after EEE (at the secondary weight) even though EEE sorts after eee (for which we need to go up to the third weight).

Now if on my system with glibc 2.27, I run:

sed -n 's/\(.*;[^[:blank:]]*\).*/\1/p' /usr/share/i18n/locales/iso14651_t1_common |
  sort -k2 | uniq -Df1

You'll notice that there are quite a few characters that have been defined with the exact same 4 weights. In particular, our ɛ has the same weights as:

<U01DD> <e>;<PCL>;<MIN>;IGNORE
<U0259> <e>;<PCL>;<MIN>;IGNORE
<U025B> <e>;<PCL>;<MIN>;IGNORE

And sure enough:

$ printf '%s\n' $'\u01DD' $'\u0259' $'\u025B' | sort -u
ǝ
$ expr ɛ = ǝ
1

That can be seen as a bug of GNU libc locales. On most other systems, locales make sure all different characters have different sorting order in the end. On GNU locales, it gets even worse, as there are thousands of characters that don't have a sorting order and end up sorting the same, causing all sorts of problems (like breaking comm, join, ls or globs having non-deterministic orders...), hence the recommendation of using LC_ALL=C to work around those issues.

As noted by @ninjalj in comments, glibc 2.28 released in August 2018 came with some improvements on that front though AFAICS, there are still some characters or collating elements defined with identical sorting order. On Ubuntu 18.10 with glibc 2.28 and in a en_GB.UTF-8 locale.

$ expr $'L\ub7' = $'L\u387'
1

(why would U+00B7 be considered equivalent as U+0387 only when combined with L/l?!).

And:

$ perl -lC -e 'for($i=0; $i<0x110000; $i++) {$i = 0xe000 if $i == 0xd800; print chr($i)}' | sort > all-chars-sorted
$ uniq -d all-chars-sorted | wc -l
4
$ uniq -D all-chars-sorted | wc -l
1061355

(still over 1 million characters (95% of the Unicode range, down from 98% in 2.27) sorting the same as other characters as their sorting order is not defined).

See also:

12
  • 3
    This is exactly what I was looking for! For completeness, what does <PCL> stand for? The others seem to be Capital, Miniscule, and Basic?
    – Draconis
    Oct 26, 2018 at 19:51
  • 4
    @Draconis, collating-symbol <PCL> # 16 particulier/peculiar Oct 26, 2018 at 20:59
  • Indeed if we put a bunch of ea and ɛa mixed together in a file we see that sort sorts all eas before ɛas.
    – Bakuriu
    Oct 27, 2018 at 9:22
  • 2
    From glibc 2.28, the codepoint should be used as a fallback for a 4th level weight, see sourceware.org/git/… sourceware.org/bugzilla/show_bug.cgi?id=14095
    – ninjalj
    Oct 27, 2018 at 10:47
  • 1
    @cat, sorry, I meant strcoll(), see edit. Oct 27, 2018 at 21:31
15

man sort:

   ***  WARNING  ***  The locale specified by the environment affects sort
   order.  Set LC_ALL=C to get the traditional sort order that uses native
   byte values.

So, try: LC_ALL=C sort file.txt

6
  • 1
    That works! But why does the default locale consider these completely separate codepoints to be the same? I'm curious why this happens.
    – Draconis
    Oct 26, 2018 at 16:36
  • @Draconis What is "the default locale"? Oct 26, 2018 at 16:39
  • @KamilMaciorowski An empty value of the environment variable; I'm not sure what locale that corresponds to.
    – Draconis
    Oct 26, 2018 at 16:44
  • 3
    @Draconis if LC_ALL is empty, sort may use other LC_* variables, LANG or some configuration files.
    – Maya
    Oct 26, 2018 at 19:43
  • 1
    LC_COLLATE is the string-sort-specific one, LANG is the extra-general one. Oct 27, 2018 at 3:16
8

The character ɛ is not equal to e, but some locales can gather these signs close together upon collation. The reason for this is language specific, but also some historical or even political background. For example most people probably expect that €uro currency comes close to Europe in dictionary.

Anyway to see what collation you are currently using run locale, the locale -a will give you the list of locales available on the system and to change collation say to C just for one sorting run LC_COLLATE=C sort file. Finally to see how different locales can sort your file try

for loc in $(locale -a)
    do echo ____"${loc}"____
    LC_COLLATE="$loc" sort file
done

Pipe the result to some greping tool to choose locale that fits your need.

8
  • This is a wonderful explanation, but the symbols seem to be considered identical, not just close together.
    – Draconis
    Oct 26, 2018 at 17:47
  • 1
    No, they're not considered identical. Add a plain ea line to the file, then with sort -u you will get both ea and ɛa in the output. The best strategy vs. collate is avoid (export LC_COLLATE=C). Otherwise, many ugly things will happen (eg. /tmp/[a-z] in bash will match /tmp/a and /tmp/A but not /tmp/Z).
    – user313992
    Oct 26, 2018 at 18:13
  • @mosvy Huh, interesting…so they are considered the same for ordering purposes but not for uniqueness purposes?
    – Draconis
    Oct 26, 2018 at 18:55
  • they're not considered the same. see here an explanation about it.
    – user313992
    Oct 26, 2018 at 19:03
  • 1
    @ninjalj, that may be fixed in the glibc fnmatch() and regexp ranges, but not in some like bash that implement its ranges by itself using strcoll(). ksh93 never had the problem because its range implementation uses strcoll() and also check the case of range ends and only match on lowercase characters if both ends are lower case. zsh ranges don't have the issue as it's done based on code point, not strcoll(). Oct 28, 2018 at 8:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .