7

I'm trying to sort a text file linewise by their Unicode values. As far as I can tell, this means numerals first, then letters, then CJK-Ideographs. However, sort (with LC_ALL=C) fails horribly at this task. Here is an excerpt from my list:

[#ゆうかりんちゃんねる]
[チ→ム♂ツナギ]
[ぞめ]
...
[サディスティックブラウニー]
[ほねとかわとがはなれるおと]
[10th Avenue Cafe]
[2nd Flush]
...
[Alstroemeria Records & Cradle]
[ALTERNATIVE]
[アルトノイラント - Altneuland]
[Amateras Records]
[セブンスヘブンAmmy's]
[anagram]
[Analyze]
...
[Z.S.G TRAXXX]
[α music]
[Яiselied]
[一人華飯スペシャル]
[七瀬屋]

It seems like sort ignores (at least sometimes) the characters it can't read, because Altneuland would indeed be between Alternative and Amateras Records. Someone suggested using msort, but it failed as well (with options -u c, -u d, and -u n, respectively).

First, why is it acting so unexpected? Second, how can I fix this?

Add:// I'm using Raspbian on a Raspberry Pi (B)

  • sorting is always dependent on locale. – jordanm Mar 13 '13 at 22:04
  • 1
    LC_ALL=C is for ASCII/7-bit, it's pretty much guaranteed to do the wrong thing for multi-byte characters. Which unicode encoding? (UTF-8, UTF-16, UTF-32, legacy UCS-x?). GNU sort with a correctly set locale is almost certainly up to the task. – mr.spuratic Mar 13 '13 at 22:31
  • @mr.spuratic, at least all of those encodings are meant to sort the same when regarded as their byte value and sorting by byte value which the C locale is meant to do. locales other than C and POSIX don't sort by byte value but follow language specific rules. – Stéphane Chazelas Mar 13 '13 at 22:43
  • @StephaneChazelas yes, of course, but I mean specifically the input file encoding, how can UTF-16 or UTF-32 sort "correctly" if the endianness is not specified? – mr.spuratic Mar 13 '13 at 23:00
  • Oh yes, you're right, I forgot about the endianness issue, not to mention the fact that utf16/ucs2 newline characters will be on two bytes, so the content would be mangled by LC_ALL=C sort. – Stéphane Chazelas Mar 13 '13 at 23:03
5

What system are you using?

LC_ALL=C sort < your-file.txt

Where your-file.txt is the text you posted in utf-8 encoding, sorts as:

[#ゆうかりんちゃんねる]
[10th Avenue Cafe]
[2nd Flush]
[ALTERNATIVE]
[Alstroemeria Records & Cradle]
[Amateras Records]
[Analyze]
[Z.S.G TRAXXX]
[anagram]
[α music]
[Яiselied]
[ぞめ]
[ほねとかわとがはなれるおと]
[アルトノイラント - Altneuland]
[サディスティックブラウニー]
[セブンスヘブンAmmy's]
[チ→ム♂ツナギ]
[一人華飯スペシャル]
[七瀬屋]

On my system (sort from GNU coreutils 8.13, Debian EGLIBC 2.13-38). Which when piped to cut -c2 | tr -d \\n | recode ..dump gives:

UCS2   Mne   Description

0023   Nb    number sign
0031   1     digit one
0032   2     digit two
0041   A     latin capital letter a
0041   A     latin capital letter a
0041   A     latin capital letter a
0041   A     latin capital letter a
005A   Z     latin capital letter z
0061   a     latin small letter a
03B1   a*    greek small letter alpha
042F   JA    cyrillic capital letter ya
305E   zo    hiragana letter zo
307B   ho    hiragana letter ho
30A2   A6    katakana letter a
30B5   Sa    katakana letter sa
30BB   Se    katakana letter se
30C1   Ti    katakana letter ti
4E00
4E03

Same on an older system with sort from GNU coreutils 7.4, EGLIBC 2.11.1-0ubuntu7.12

| improve this answer | |
  • I'm using a Raspberry Pi. I'll later try and see what your command results in – MechMK1 Mar 14 '13 at 9:51
  • No, on Ubuntu with sort 8.12 I get the exact same result as on my Raspberry Pi – MechMK1 Mar 14 '13 at 12:05
2

The question has been solved. Appearently LC_ALL=C is not enough. I need to write export LC_ALL=C in order to be recognized by sort. This is my bad, but should be noted for future unicode-sorters

| improve this answer | |
  • 1
    Well, yes var=value sets a shell variable, so is not going to affect any other command unless that shell variable is marked for exporting. – Stéphane Chazelas Mar 14 '13 at 12:44
  • 3
    No, you should not. You just need to add LC_ALL=C before sort command. – gaRex Nov 14 '18 at 5:18
  • 2
    Compare echo 🥚 🐔 | tr ' ' '\n' | sort – gaRex Nov 14 '18 at 5:18
  • 3
    .. with echo 🥚 🐔 | tr ' ' '\n' | LC_ALL=C sort – gaRex Nov 14 '18 at 5:19
  • Thank you very much for your comment. This question was asked by me 5 and a half years ago and in the meantime I have learned a lot more about how shells work. – MechMK1 Nov 16 '18 at 15:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.