7

I have a OSX machine where sort runs GNU sort from coreutils 8.26 (installed from Homebrew), and a Linux machine where sort runs GNU sort from coreutils 8.25.

On the Mac:

mac$ echo -e "{1\n2" | sort
2
{1

While on Linux:

linux$ echo -e "{1\n2" | sort
{1
2

I'm aware that sort depends on the locale. I ran locale on the Linux machine, prepended each line of output with export and ran the resulting lines on the OSX machine before running (in the same terminal) the sort command again, which gave the same output as before.

I noticed, however, that running locale on the Mac doesn't show all of the lines which appear on Linux, and I'm not sure if this is related.

The locale on Linux:

linux$ locale
LANG=en_CA.UTF-8
LANGUAGE=en_CA:en
LC_CTYPE="en_CA.UTF-8"
LC_NUMERIC="en_CA.UTF-8"
LC_TIME="en_CA.UTF-8"
LC_COLLATE="en_CA.UTF-8"
LC_MONETARY="en_CA.UTF-8"
LC_MESSAGES="en_CA.UTF-8"
LC_PAPER="en_CA.UTF-8"
LC_NAME="en_CA.UTF-8"
LC_ADDRESS="en_CA.UTF-8"
LC_TELEPHONE="en_CA.UTF-8"
LC_MEASUREMENT="en_CA.UTF-8"
LC_IDENTIFICATION="en_CA.UTF-8"
LC_ALL=en_CA.UTF-8

And locale on OSX:

mac$ locale
LANG="en_CA.UTF-8"
LC_COLLATE="en_CA.UTF-8"
LC_CTYPE="en_CA.UTF-8"
LC_MESSAGES="en_CA.UTF-8"
LC_MONETARY="en_CA.UTF-8"
LC_NUMERIC="en_CA.UTF-8"
LC_TIME="en_CA.UTF-8"
LC_ALL="en_CA.UTF-8"

I've found that if I set LC_ALL=C on both machines, they both sort 2 before {1. But if I set LC_ALL=en_CA.UTF-8 on both machines I have the differing output as above. Same if I set LC_ALL=en_CA.utf8 on both machines. (locale -a lists en_CA.utf8 on the Linux machine but en_CA.UTF-8 on the OSX machine.)

Any idea what is going on here?

  • 2
    Try using only locales that actually exist. Otherwise it defaults to C, which you probably don't want – Fox May 3 '17 at 0:05
  • @Fox, I don't know what you're telling me. I said that locale -a lists the Canadian English UTF-8 locale differently on each machine, and that I'd tried both variants on both machines, and that this made no difference. I only have the Linux machine at my fingertips right now, and have tried the sort operation with every locale installed (a bunch of English variants plus C and POSIX) and {1 always sorts before 2 here. – tremby May 4 '17 at 1:19
  • 1
    It's not entirely clear from your Q which combinations you've chosen. Having OS X as en_CA.UTF-8 and Linux as en_CA.utf8 should be consistent, unless the collation rules are specified differently for some reason – Fox May 4 '17 at 1:22
  • I take back what I said at the end of my last comment; my loop was wrong. In fact on Linux it sorts {1 before 2 in all locales except the C variants and POSIX, which sort it the other way around. Further, it doesn't make a difference whether I use .utf8 or .UTF-8, and when running locale-gen the .UTF-8 version is displayed, even though locale -a shows the .utf8 version. This suggests they'll work the same way anyway. I'll test the Mac side of things again tomorrow. Thanks. – tremby May 4 '17 at 1:30
  • 3
    This answer suggests, unfortunately without visible references, that the locales in BSD (and thus macOS) are "somewhat broken compared to those in Linux" – Fox May 6 '17 at 0:59
3

I did some digging on the same problem the other day, so let me share a technical answer.


On macOS, /usr/share/locale/en_US.UTF-8/LC_COLLATE (or en_CA.UTF-8, same thing) is a symlink to /usr/share/locale/la_LN.US-ASCII/LC_COLLATE, which is generated from la_LN.US-ASCII.src with colldef. Here's the entirety of la_LN.US-ASCII.src:

# ASCII
#
# $FreeBSD: src/share/colldef/la_LN.US-ASCII.src,v 1.2 1999/08/28 00:59:47 peter Exp $
#
order \
    \x00;...;\xff

You can verify that the binary LC_COLLATE file is indeed generated from la_LN.US-ASCII.src by verifying checksums:

$ colldef -o /dev/stdout usr-share-locale.tproj/colldef/la_LN.US-ASCII.src | sha256sum
9ec9b40c837860a43eb3435d7a9cc8235e66a1a72463d11e7f750500cabb5b78  -

$ sha256sum </usr/share/locale/en_US.UTF-8/LC_COLLATE
9ec9b40c837860a43eb3435d7a9cc8235e66a1a72463d11e7f750500cabb5b78  -

The ruleset is easily understandable: just compare the byte values one by one. So the collation rules for en_US.UTF-8 are the same as the POSIX locale (aka C locale). { is 0x7B, 2 is 0x32, so { comes after 2.

This ruleset is an artifact of FreeBSD 5, synced into Mac OS X 10.3 Panther. See colldef directory in FreeBSD 5.0.0 source tree. It never changed on OS X / macOS since.


On Linux, locale programs and data are part of glibc. See glibc localedata/locales tree, or /usr/share/i18n/locales on Debian/Ubuntu. If you inspect /usr/share/i18n/locales/en_US, you'll see that it pulls in iso14651_t1_common for LC_COLLATE rules. So it follows ISO 14651 rules for collation.


There are more details in the blog post: https://blog.zhimingwang.org/macos-lc_collate-hunt.

| improve this answer | |
  • I wonder if you might add a conclusion -- if I'm understanding correctly you're saying the same as in @Fox's comment above, that basically the locales on BSD and therefore on MacOS are "somewhat broken compared to those in Linux". I wonder also if there's anything that can be done. Can it be "fixed"? – tremby Jun 8 at 2:03
  • You may reach that conclusion, but "broken" seems subjective since AFAIK POSIX spec doesn't mandate any locale other than C at all, let alone actual collation rules, and I haven't read Unicode specs so I can't say if Unicode mandates ISO 14651. It's unsophisticated, that much I can say, objectively. – 4ae1e1 Jun 8 at 5:33
  • @tremby As for can it be "fixed"? I suppose you can pluck the locale source files from the current FreeBSD tree github.com/freebsd/freebsd/tree/release/12.1.0/share/colldef compile and replace /usr/share/locale/*/LC_COLLATE, but you could very well break unknown things, so I wouldn't recommend it. – 4ae1e1 Jun 8 at 5:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.