What value for LANG should I use for "sort -u“ correctly handle Chinese characters?

Question

• The context
• Issues found
  • Issue no. 1
  • Issue no. 2
  • Issue no. 3
• The question

The context

Today, I wanted to keep the unique lines in a file containing Chinese characters. I decided to use the sort utility because I'm familiar with that tool and deleting repeated lines in a file is as easy as using the -u flag. I learnt that I needed to change the locale to make sort work correctly with Chinese characters. I noticed that using different locales made sort have different behaviors. In this post, I showed what I found.

I'm aware that the task of deleting repeated lines in a file can be done using multiple tools/programming languages. While I thank anyone that suggests a tool for doing that task, I'm more interested in knowing more about locales and how they affect Unix utilities.

Issues found

Issue no. 1

The following is the locale of my system.

locale

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Consider the following file named main.txt

䔍
䏝

If I try to sort it using en_US.UTF-8 as my $LANG. I lose line no. 2

sort -u main.txt

䔍

I have solved this issue by setting LANG to zh_CN.UTF-8.

export LANG=zh_CN.UTF-8
sort -u main.txt

䔍
䏝

I created a question about this issue in Stack Overflow. Someone suggested me changing the locale and it seemed to work, I thought I had solved the problem, but then I found Issue no. 2 and Issue no. 3 (described below).

Issue no. 2

Now, let's suppose that we add two lines to our file main.txt

䔍
䏝
𠵇
𠳐

If I execute sort -u on it while having LANG equal to zh_CN.UTF-8, I lose line no. 4.

export LANG=zh_CN.UTF-8
sort -u main.txt

䔍
𠵇
䏝

I tried different locales that started with zh.CN since the characters are Chinese characters. The uncommented locales are the ones that I tried using.

grep '^#\?zh' /etc/locale.gen

zh_CN.GB18030 GB18030  
zh_CN.GBK GBK  
zh_CN.UTF-8 UTF-8  
zh_CN GB2312  
#zh_HK.UTF-8 UTF-8  
#zh_HK BIG5-HKSCS  
#zh_SG.UTF-8 UTF-8  
#zh_SG.GBK GBK  
#zh_SG GB2312  
#zh_TW.EUC-TW EUC-TW  
#zh_TW.UTF-8 UTF-8  
#zh_TW BIG5  

I noticed that sort -u works as expected when setting LANG set to zh_CN.GBK or zh_CN.GB18030. I solved this issue by using either zh_CN.GBK or zh_CN.GB18030.

Here's the proof that it works as expected when setting LANG to zh_CN.GBK.

export LANG=zh_CN.GBK
sort -u main.txt

䔍
𠵇
𠳐
䏝

Here's the proof that it works as expected when setting LANG to zh_CN.GB18030.

export LANG=zh_CN.GB18030
sort -u main.txt

䔍
𠵇
𠳐
䏝

Issue no. 3

Let's suppose that we add two more lines to our file main.txt

䔍
䏝
𠵇
𠳐
书
乙

If I use zh_CN.GB18030 or zh_CN.GBK, I lose line no. 6.

export LANG=zh_CN.GB18030
sort -u main.txt

书
䔍
𠵇
𠳐
䏝

export LANG=zh_CN.GBK
sort -u main.txt

书
䔍
𠵇
𠳐
䏝

The question

Which value for $LANG should I use for making sort actually keep the unique lines in the following file?

䔍
䏝
𠵇
𠳐
书
乙

I previously showed that using zh_CN.GB18030, zh_CN.GBK, zh_CN.UTF-8 or zh_CN doesn't seem to work.

Answer 1

sort -u outputs one of each set of lines that collate the same (have identical sorting order) in the locale.

On GNU systems (systems using the GNU libc), in most locales, many characters¹ have an undefined sorting order which means they end up sorting the same, and there are also characters that are explicitly defined to have the same sorting order.

When bytes cannot be decoded into text, they will often also be ignored or considered to have the same sorting order².

So for sort -u to give you unique lines based on byte-to-byte equality comparison whatever encoding the text is written in, you need a locale where every byte can be decoded into a character and a locale with a total order.

The easiest is to use the C locale where there's a one byte == one character mapping (some bytes possibly leading to undefined characters) and sorting is based on the byte value (on ASCII-based systems at least) and is the only locale which you're guaranteed to find on any given system.

In your case, to understand why zh_CN gives you slightly better behaviour, you can have a look at the locale definitions typically in /usr/share/i18n/locale on GNU systems where for zh_CN you find:

LC_COLLATE
copy "iso14651_t1_pinyin"
END LC_COLLATE

Where iso14651_t1_pinyin has:

copy "iso14651_t1_common"

And then specifies ordering for many additional Chinese characters, while en_US only uses iso14651_t1_common.

If you set LC_CTYPE (via LANG) to zh_CN.GB18030 or zh_CN.GBK and try to process a file encoded in UTF-8, sort will typically fail to decode the lines into text as the bytes of UTF-8 encoded text typically don't form valid GB18030 encoded text and you'll make the problem a lot worse.

Here, if you want to get unique (as per byte-to-byte comparison) lines but also want to get a "properly" sorted output in your locale, you can do:

LC_ALL=C sort -u your-file | sort

Where the first sort removes duplicates and the second sorts the remaining lines as per your locale collation order.

We use LC_ALL instead of LANG or LC_CTYPE+LC_COLLATE because that's the variable that overrides it all (LC_ALL has precedence over LC_individual_setting which has precedence over LANG) so it still works if LANG or LC_COLLATE is otherwise also set in the environment.

In theory, you could also do:

LC_ALL= LC_CTYPE=C LC_COLLATE=C sort

so as to set only the "character type" and "collation" settings, leaving the other locale categories alone (covered by LANG or LC_xxx), but the remaining one used by sort is LC_MESSAGES used in error messages for instance, and if you set LC_CTYPE=C, only US-English messages can be displayed properly anyway as it's only got ASCII characters. For example:

$ LC_MESSAGES=zh_CN LC_CTYPE=C sort --version
sort (GNU coreutils) 9.1
Copyright (C) 2022 Free Software Foundation, Inc.
??? GPLv3+:GNU ???????? 3 ?????? <https://gnu.org/licenses/gpl.html>?
????????:??????????????
?????????,????????

? Mike Haertel ? Paul Eggert ???

You'll find more details at:

• What does "LC_ALL=C" do?
• Using uniq on unicode text
• What is the difference between "sort -u" and "sort | uniq"?

---

^{¹ Strictly speaking, the primary unit is not character but collating element which can be made of more that one character like ch that sorts between h and i in some Czech locales.}

^{² Beware some sort implementations (not GNU sort) would also choke on NUL characters or excessively long lines.}