8

I'm trying to sort a text file linewise by their Unicode values. As far as I can tell, this means numerals first, then letters, then CJK-Ideographs. However, sort (with LC_ALL=C) fails horribly at this task. Here is an excerpt from my list:

[#ゆうかりんちゃんねる]
[チ→ム♂ツナギ]
[ぞめ]
...
[サディスティックブラウニー]
[ほねとかわとがはなれるおと]
[10th Avenue Cafe]
[2nd Flush]
...
[Alstroemeria Records & Cradle]
[ALTERNATIVE]
[アルトノイラント - Altneuland]
[Amateras Records]
[セブンスヘブンAmmy's]
[anagram]
[Analyze]
...
[Z.S.G TRAXXX]
[α music]
[Яiselied]
[一人華飯スペシャル]
[七瀬屋]

It seems like sort ignores (at least sometimes) the characters it can't read, because Altneuland would indeed be between Alternative and Amateras Records. Someone suggested using msort, but it failed as well (with options -u c, -u d, and -u n, respectively).

First, why is it acting so unexpected? Second, how can I fix this?

Add:// I'm using Raspbian on a Raspberry Pi (B)

Improve this question
5
  • sorting is always dependent on locale.
    – jordanm
    Commented Mar 13, 2013 at 22:04
  • 1
    LC_ALL=C is for ASCII/7-bit, it's pretty much guaranteed to do the wrong thing for multi-byte characters. Which unicode encoding? (UTF-8, UTF-16, UTF-32, legacy UCS-x?). GNU sort with a correctly set locale is almost certainly up to the task.
    – mr.spuratic
    Commented Mar 13, 2013 at 22:31
  • @mr.spuratic, at least all of those encodings are meant to sort the same when regarded as their byte value and sorting by byte value which the C locale is meant to do. locales other than C and POSIX don't sort by byte value but follow language specific rules.
    – Stéphane Chazelas
    Commented Mar 13, 2013 at 22:43
  • @StephaneChazelas yes, of course, but I mean specifically the input file encoding, how can UTF-16 or UTF-32 sort "correctly" if the endianness is not specified?
    – mr.spuratic
    Commented Mar 13, 2013 at 23:00
  • Oh yes, you're right, I forgot about the endianness issue, not to mention the fact that utf16/ucs2 newline characters will be on two bytes, so the content would be mangled by LC_ALL=C sort.
    – Stéphane Chazelas
    Commented Mar 13, 2013 at 23:03

2 Answers 2

Reset to default
7

What system are you using?

LC_ALL=C sort < your-file.txt

Where your-file.txt is the text you posted in utf-8 encoding, sorts as:

[#ゆうかりんちゃんねる]
[10th Avenue Cafe]
[2nd Flush]
[ALTERNATIVE]
[Alstroemeria Records & Cradle]
[Amateras Records]
[Analyze]
[Z.S.G TRAXXX]
[anagram]
[α music]
[Яiselied]
[ぞめ]
[ほねとかわとがはなれるおと]
[アルトノイラント - Altneuland]
[サディスティックブラウニー]
[セブンスヘブンAmmy's]
[チ→ム♂ツナギ]
[一人華飯スペシャル]
[七瀬屋]

On my system (sort from GNU coreutils 8.13, Debian EGLIBC 2.13-38). Which when piped to cut -c2 | tr -d \\n | recode ..dump gives:

UCS2   Mne   Description

0023   Nb    number sign
0031   1     digit one
0032   2     digit two
0041   A     latin capital letter a
0041   A     latin capital letter a
0041   A     latin capital letter a
0041   A     latin capital letter a
005A   Z     latin capital letter z
0061   a     latin small letter a
03B1   a*    greek small letter alpha
042F   JA    cyrillic capital letter ya
305E   zo    hiragana letter zo
307B   ho    hiragana letter ho
30A2   A6    katakana letter a
30B5   Sa    katakana letter sa
30BB   Se    katakana letter se
30C1   Ti    katakana letter ti
4E00
4E03

Same on an older system with sort from GNU coreutils 7.4, EGLIBC 2.11.1-0ubuntu7.12

Improve this answer
2
  • I'm using a Raspberry Pi. I'll later try and see what your command results in
    – MechMK1
    Commented Mar 14, 2013 at 9:51
  • No, on Ubuntu with sort 8.12 I get the exact same result as on my Raspberry Pi
    – MechMK1
    Commented Mar 14, 2013 at 12:05
3

The question has been solved. Appearently LC_ALL=C is not enough. I need to write export LC_ALL=C in order to be recognized by sort. This is my bad, but should be noted for future unicode-sorters

Improve this answer
5
  • 1
    Well, yes var=value sets a shell variable, so is not going to affect any other command unless that shell variable is marked for exporting.
    – Stéphane Chazelas
    Commented Mar 14, 2013 at 12:44
  • 3
    No, you should not. You just need to add LC_ALL=C before sort command.
    – gaRex
    Commented Nov 14, 2018 at 5:18
  • 2
    Compare echo 🥚 🐔 | tr ' ' '\n' | sort
    – gaRex
    Commented Nov 14, 2018 at 5:18
  • 3
    .. with echo 🥚 🐔 | tr ' ' '\n' | LC_ALL=C sort
    – gaRex
    Commented Nov 14, 2018 at 5:19
  • Thank you very much for your comment. This question was asked by me 5 and a half years ago and in the meantime I have learned a lot more about how shells work.
    – MechMK1
    Commented Nov 16, 2018 at 15:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .