- The context
- Issues found
- Issue no. 1
- Issue no. 2
- Issue no. 3
- The question
The context
Today, I wanted to keep the unique lines in a file containing Chinese characters. I decided to use the sort
utility because I'm familiar with that tool and deleting repeated lines in a file is as easy as using the -u
flag. I learnt that I needed to change the locale to make sort
work correctly with Chinese characters. I noticed that using different locales made sort
have different behaviors. In this post, I showed what I found.
I'm aware that the task of deleting repeated lines in a file can be done using multiple tools/programming languages. While I thank anyone that suggests a tool for doing that task, I'm more interested in knowing more about locales and how they affect Unix utilities.
Issues found
Issue no. 1
The following is the locale of my system.
locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
Consider the following file named main.txt
䔍
䏝
If I try to sort it using en_US.UTF-8
as my $LANG
. I lose line no. 2
sort -u main.txt
䔍
I have solved this issue by setting LANG
to zh_CN.UTF-8
.
export LANG=zh_CN.UTF-8
sort -u main.txt
䔍
䏝
I created a question about this issue in Stack Overflow. Someone suggested me changing the locale and it seemed to work, I thought I had solved the problem, but then I found Issue no. 2 and Issue no. 3 (described below).
Issue no. 2
Now, let's suppose that we add two lines to our file main.txt
䔍
䏝
𠵇
𠳐
If I execute sort -u
on it while having LANG
equal to zh_CN.UTF-8
, I lose line no. 4.
export LANG=zh_CN.UTF-8
sort -u main.txt
䔍
𠵇
䏝
I tried different locales that started with zh.CN
since the characters are Chinese characters. The uncommented locales are the ones that I tried using.
grep '^#\?zh' /etc/locale.gen
zh_CN.GB18030 GB18030
zh_CN.GBK GBK
zh_CN.UTF-8 UTF-8
zh_CN GB2312
#zh_HK.UTF-8 UTF-8
#zh_HK BIG5-HKSCS
#zh_SG.UTF-8 UTF-8
#zh_SG.GBK GBK
#zh_SG GB2312
#zh_TW.EUC-TW EUC-TW
#zh_TW.UTF-8 UTF-8
#zh_TW BIG5
I noticed that sort -u
works as expected when setting LANG
set to zh_CN.GBK
or zh_CN.GB18030
. I solved this issue by using either zh_CN.GBK
or zh_CN.GB18030
.
Here's the proof that it works as expected when setting LANG
to zh_CN.GBK
.
export LANG=zh_CN.GBK
sort -u main.txt
䔍
𠵇
𠳐
䏝
Here's the proof that it works as expected when setting LANG
to zh_CN.GB18030
.
export LANG=zh_CN.GB18030
sort -u main.txt
䔍
𠵇
𠳐
䏝
Issue no. 3
Let's suppose that we add two more lines to our file main.txt
䔍
䏝
𠵇
𠳐
书
乙
If I use zh_CN.GB18030
or zh_CN.GBK
, I lose line no. 6.
export LANG=zh_CN.GB18030
sort -u main.txt
书
䔍
𠵇
𠳐
䏝
export LANG=zh_CN.GBK
sort -u main.txt
书
䔍
𠵇
𠳐
䏝
The question
Which value for $LANG
should I use for making sort
actually keep the unique lines in the following file?
䔍
䏝
𠵇
𠳐
书
乙
I previously showed that using zh_CN.GB18030
, zh_CN.GBK
, zh_CN.UTF-8
or zh_CN
doesn't seem to work.