Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm trying to sort a text file linewise by their Unicode values. As far as I can tell, this means numerals first, then letters, then CJK-Ideographs. However, sort (with LC_ALL=C) fails horribly at this task. Here is an excerpt from my list:

[#ゆうかりんちゃんねる]
[チ→ム♂ツナギ]
[ぞめ]
...
[サディスティックブラウニー]
[ほねとかわとがはなれるおと]
[10th Avenue Cafe]
[2nd Flush]
...
[Alstroemeria Records & Cradle]
[ALTERNATIVE]
[アルトノイラント - Altneuland]
[Amateras Records]
[セブンスヘブンAmmy's]
[anagram]
[Analyze]
...
[Z.S.G TRAXXX]
[α music]
[Яiselied]
[一人華飯スペシャル]
[七瀬屋]

It seems like sort ignores (at least sometimes) the characters it can't read, because Altneuland would indeed be between Alternative and Amateras Records. Someone suggested using msort, but it failed as well (with options -u c, -u d, and -u n, respectively).

First, why is it acting so unexpected? Second, how can I fix this?

Add:// I'm using Raspbian on a Raspberry Pi (B)

share|improve this question
    
sorting is always dependent on locale. –  jordanm Mar 13 '13 at 22:04
1  
LC_ALL=C is for ASCII/7-bit, it's pretty much guaranteed to do the wrong thing for multi-byte characters. Which unicode encoding? (UTF-8, UTF-16, UTF-32, legacy UCS-x?). GNU sort with a correctly set locale is almost certainly up to the task. –  mr.spuratic Mar 13 '13 at 22:31
    
@mr.spuratic, at least all of those encodings are meant to sort the same when regarded as their byte value and sorting by byte value which the C locale is meant to do. locales other than C and POSIX don't sort by byte value but follow language specific rules. –  Stephane Chazelas Mar 13 '13 at 22:43
    
@StephaneChazelas yes, of course, but I mean specifically the input file encoding, how can UTF-16 or UTF-32 sort "correctly" if the endianness is not specified? –  mr.spuratic Mar 13 '13 at 23:00
    
Oh yes, you're right, I forgot about the endianness issue, not to mention the fact that utf16/ucs2 newline characters will be on two bytes, so the content would be mangled by LC_ALL=C sort. –  Stephane Chazelas Mar 13 '13 at 23:03
add comment

2 Answers

up vote 1 down vote accepted

What system are you using?

LC_ALL=C sort < your-file.txt

Where your-file.txt is the text you posted in utf-8 encoding, sorts as:

[#ゆうかりんちゃんねる]
[10th Avenue Cafe]
[2nd Flush]
[ALTERNATIVE]
[Alstroemeria Records & Cradle]
[Amateras Records]
[Analyze]
[Z.S.G TRAXXX]
[anagram]
[α music]
[Яiselied]
[ぞめ]
[ほねとかわとがはなれるおと]
[アルトノイラント - Altneuland]
[サディスティックブラウニー]
[セブンスヘブンAmmy's]
[チ→ム♂ツナギ]
[一人華飯スペシャル]
[七瀬屋]

On my system (sort from GNU coreutils 8.13, Debian EGLIBC 2.13-38). Which when piped to cut -c2 | tr -d \\n | recode ..dump gives:

UCS2   Mne   Description

0023   Nb    number sign
0031   1     digit one
0032   2     digit two
0041   A     latin capital letter a
0041   A     latin capital letter a
0041   A     latin capital letter a
0041   A     latin capital letter a
005A   Z     latin capital letter z
0061   a     latin small letter a
03B1   a*    greek small letter alpha
042F   JA    cyrillic capital letter ya
305E   zo    hiragana letter zo
307B   ho    hiragana letter ho
30A2   A6    katakana letter a
30B5   Sa    katakana letter sa
30BB   Se    katakana letter se
30C1   Ti    katakana letter ti
4E00
4E03

Same on an older system with sort from GNU coreutils 7.4, EGLIBC 2.11.1-0ubuntu7.12

share|improve this answer
    
I'm using a Raspberry Pi. I'll later try and see what your command results in –  MechMK1 Mar 14 '13 at 9:51
    
No, on Ubuntu with sort 8.12 I get the exact same result as on my Raspberry Pi –  MechMK1 Mar 14 '13 at 12:05
add comment

The question has been solved. Appearently LC_ALL=C is not enough. I need to write export LC_ALL=C in order to be recognized by sort. This is my bad, but should be noted for future unicode-sorters

share|improve this answer
1  
Well, yes var=value sets a shell variable, so is not going to affect any other command unless that shell variable is marked for exporting. –  Stephane Chazelas Mar 14 '13 at 12:44
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.