Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

What is going on in the following code snippet? I'm not getting my expected output.

I'd think it was a bug, but it happens for 2 different programs (uniq and sort), so I suspect it is something to do with... well, I don't know what.. hence the question.

The first 3 (of 4) examples work, but the 4th fails!.

I would expect the same behaviour for any and all characters.
ie. to print out 2 lines (from the 3 lines of input)... but in the 4th case, I only get 1 line (for both sort -u and uniq); the two identical lins just vanish!

I've converted the output '\n' to space for compactness of view.

I'm using uniq and sort from (GNU coreutils) 7.4 ... running on Ubuntu 10.04.3 LTS desktop.

The script:

{
  locale -k LC_COLLATE
  echo
  for c1 in x 〼 ;do 
    for c2 in z 〇 ;do 
      echo -n "asis   : "; echo -e "$c1\n$c2\n$c2"          |tr '\n' ' ';echo
      echo -n "uniq   : "; echo -e "$c1\n$c2\n$c2" |uniq    |tr '\n' ' ';echo
      echo -n "sort -u: "; echo -e "$c1\n$c2\n$c2" |sort -u |tr '\n' ' ';echo
      echo
    done
    echo
  done
}

The output:

collate-nrules=4
collate-rulesets=""
collate-symb-hash-sizemb=2081
collate-codeset="UTF-8"

asis   : x z z 
uniq   : x z 
sort -u: x z 

asis   : x 〇 〇 
uniq   : x 〇 
sort -u: 〇 x 


asis   : 〼 z z 
uniq   : 〼 z 
sort -u: 〼 z 

asis   : 〼 〇 〇 
uniq   : 〼 
sort -u: 〼 

# In the last example (of 4) where did the '〇' go? .. U+3007 IDEOGRAPHIC NUMBER ZERO
#
share|improve this question
    
Please note .. To make it quite clear. sort alone (without the -u option) ... does not gobble up characters.. What goes in, comes out... However, as may be expected by Gilles explanation of the "exotic" unicode chars having the same canonical value, these characters do not get sorted, other than that they are output as an unsorted FIFO group to the "top" of the sort's output... So there are really two issues here: 1. The characters do not get sorted as might be "naively" expected, and 2. The "unique" feature of both sort and uniq lose data (in some cases). –  Peter.O Jul 23 '11 at 0:35
    
Update: As mentioned by Gilles (when locale-specific sorting is not essential, and character order is suitable), sort -u and uniq work fine with: LC_COLLATE=C; echo -e "〼\n〇\n〇" |sort -u (or |uniq ) –  Peter.O Jul 23 '11 at 4:07

2 Answers 2

up vote 3 down vote accepted

Short version: collation doesn't really work in command line utilities.

Longer version: the underlying function to compare two strings is strcoll. The description isn't very helpful, but the conceptual method of operation is to convert both strings to a canonical form, and then compare the two canonical forms. The function strxfrm constructs this canonical form.

Let's observe the canonical forms of a few strings (with GNU libc, under Debian squeeze):

$ export LC_ALL=en_US.UTF-8
$ perl -C255 -MPOSIX -le 'print "$_ ", unpack("h*", strxfrm($_)) foreach @ARGV' b a A à 〼 〇
b d010801020
a c010801020
A c010801090
à 101010102c6b
〼 101010102c6b102c6b102c6b
〇 101010102c6b102c6b102c6b

As you can see, 〼 and 〇 have the same canonical form. I think that's because these characters are not mentioned in the collation tables of the en_US.UTF-8 locale. They are, however, present in a Japanese locale.

$ export LC_ALL=ja_JP.UTF-8
$ perl -C255 -MPOSIX -le 'print "$_ ", unpack("h*", strxfrm($_)) foreach @ARGV' 〼 〇 
〼 303030
〇 3c9b

The source code for the locale data (in Debian squeeze) is in /usr/share/i18n/locales/en_US, which includes /usr/share/i18n/locales/iso14651_t1_common. This file doesn't have an entry for U3007 or U303C, nor are they included in any range that I can find.

I'm not familiar with the rules to build the collation order, but from what I understand, the relevant phrasing is

The symbol UNDEFINED shall be interpreted as including all coded character set values not specified explicitly or via the ellipsis symbol. (…) If no UNDEFINED symbol is specified, and the current coded character set contains characters not specified in this section, the utility shall issue a warning message and place such characters at the end of the character collation order.

It looks like Glibc is instead ignoring characters that aren't specified. I don't know if there's a flaw of my understanding of the POSIX spec, if I missed something in Glibc's locale definition, or if there's a bug in the Glibc locale compiler.

share|improve this answer
    
@Gilles: Thanks for the informative and detailed explanation.. It makes some sense now, but I'm left wondering, how to "safely" use sort.. I'm not after a particularly "locale sensitive" sort, so any rough sort would do... Is there a quick workaround for this? ... and I'll gradually get the hang of this, but it won't happen 'overnight'... eg.. my /usr/share/i18n/charmaps/UTF-8 contains references to both the characters in question, but being in this UTF-8 definition(?) doesn't seem to help... Oh well, what would life be like without its little mysteries. :) ... –  Peter.O Jul 22 '11 at 18:30
1  
@fred charmaps/UTF-8 doesn't say anything about collation, it's locales/en_US that matters. The first rule of LC_COLLATE is: don't use LC_COLLATE. In the C (= POSIX) locale, collation is reasonable (based strictly on numerical character values). –  Gilles Jul 22 '11 at 19:10
    
The sorting and unique aspect work fine when preceded by LC_COLLATE=C... thanks... –  Peter.O Jul 23 '11 at 4:09

To "safely" sort Unicode strings, maybe have a look at msort:

[...] Msort provides greater flexibility in selecting key fields, more comparison types, the ability to use collation rules from different locales on different keys, the ability to handle numbers in non-Western number systems, and a variety of other options lacking in GNU sort and BSD sort. Whereas msort understands Unicode, GNU sort and BSD sort do not. [...]

http://www.billposer.org/Software/msort.html

share|improve this answer
    
@til: Thanks for making me aware of msort. The optional GUI makes the introduction a bit easier to get a feel of what's on offer. Being able to copy the generated command is very handy... And yes, it does sort the unicode characters, but (don't you just love those "buts" :)... but it doesn't have a unique option :( ...as mentioned on the link you posted: Capabilities of GNU sort and BSD sort lacking in msort are the ability to merge files without sorting them (the --merge option) and the ability to emit only the first of an equal run (the --unique option)... The sort works though :) –  Peter.O Jul 23 '11 at 0:55

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.