Tell me more ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 2 down vote favorite

I'm trying to sort a text file linewise by their Unicode values. As far as I can tell, this means numerals first, then letters, then CJK-Ideographs. However, sort (with LC_ALL=C) fails horribly at this task. Here is an excerpt from my list:

[#ゆうかりんちゃんねる]
[チ→ム♂ツナギ]
[ぞめ]
...
[サディスティックブラウニー]
[ほねとかわとがはなれるおと]
[10th Avenue Cafe]
[2nd Flush]
...
[Alstroemeria Records & Cradle]
[ALTERNATIVE]
[アルトノイラント - Altneuland]
[Amateras Records]
[セブンスヘブンAmmy's]
[anagram]
[Analyze]
...
[Z.S.G TRAXXX]
[α music]
[Яiselied]
[一人華飯スペシャル]
[七瀬屋]

It seems like sort ignores (at least sometimes) the characters it can't read, because Altneuland would indeed be between Alternative and Amateras Records. Someone suggested using msort, but it failed as well (with options -u c, -u d, and -u n, respectively).

First, why is it acting so unexpected? Second, how can I fix this?

Add:// I'm using Raspbian on a Raspberry Pi (B)

share|improve this question
sorting is always dependent on locale. – jordanm Mar 13 at 22:04
1  
LC_ALL=C is for ASCII/7-bit, it's pretty much guaranteed to do the wrong thing for multi-byte characters. Which unicode encoding? (UTF-8, UTF-16, UTF-32, legacy UCS-x?). GNU sort with a correctly set locale is almost certainly up to the task. – mr.spuratic Mar 13 at 22:31
@mr.spuratic, at least all of those encodings are meant to sort the same when regarded as their byte value and sorting by byte value which the C locale is meant to do. locales other than C and POSIX don't sort by byte value but follow language specific rules. – Stephane Chazelas Mar 13 at 22:43
@StephaneChazelas yes, of course, but I mean specifically the input file encoding, how can UTF-16 or UTF-32 sort "correctly" if the endianness is not specified? – mr.spuratic Mar 13 at 23:00
Oh yes, you're right, I forgot about the endianness issue, not to mention the fact that utf16/ucs2 newline characters will be on two bytes, so the content would be mangled by LC_ALL=C sort. – Stephane Chazelas Mar 13 at 23:03

2 Answers

up vote 1 down vote accepted

What system are you using?

LC_ALL=C sort < your-file.txt

Where your-file.txt is the text you posted in utf-8 encoding, sorts as:

[#ゆうかりんちゃんねる]
[10th Avenue Cafe]
[2nd Flush]
[ALTERNATIVE]
[Alstroemeria Records & Cradle]
[Amateras Records]
[Analyze]
[Z.S.G TRAXXX]
[anagram]
[α music]
[Яiselied]
[ぞめ]
[ほねとかわとがはなれるおと]
[アルトノイラント - Altneuland]
[サディスティックブラウニー]
[セブンスヘブンAmmy's]
[チ→ム♂ツナギ]
[一人華飯スペシャル]
[七瀬屋]

On my system (sort from GNU coreutils 8.13, Debian EGLIBC 2.13-38). Which when piped to cut -c2 | tr -d \\n | recode ..dump gives:

UCS2   Mne   Description

0023   Nb    number sign
0031   1     digit one
0032   2     digit two
0041   A     latin capital letter a
0041   A     latin capital letter a
0041   A     latin capital letter a
0041   A     latin capital letter a
005A   Z     latin capital letter z
0061   a     latin small letter a
03B1   a*    greek small letter alpha
042F   JA    cyrillic capital letter ya
305E   zo    hiragana letter zo
307B   ho    hiragana letter ho
30A2   A6    katakana letter a
30B5   Sa    katakana letter sa
30BB   Se    katakana letter se
30C1   Ti    katakana letter ti
4E00
4E03

Same on an older system with sort from GNU coreutils 7.4, EGLIBC 2.11.1-0ubuntu7.12

share|improve this answer
I'm using a Raspberry Pi. I'll later try and see what your command results in – David Stockinger Mar 14 at 9:51
No, on Ubuntu with sort 8.12 I get the exact same result as on my Raspberry Pi – David Stockinger Mar 14 at 12:05
up vote 0 down vote

The question has been solved. Appearently LC_ALL=C is not enough. I need to write export LC_ALL=C in order to be recognized by sort. This is my bad, but should be noted for future unicode-sorters

share|improve this answer
1  
Well, yes var=value sets a shell variable, so is not going to affect any other command unless that shell variable is marked for exporting. – Stephane Chazelas Mar 14 at 12:44

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.