Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Trying to determine all characters in a file.

The file sample consists of:

a eɪ
abandon əˈbændən
ability əˈbɪləti
able ˈeɪbəl
able ˈeɪbl
abortion əˈbɔrʃən
abortion əˈbɔrʃn
about əˈbaʊt
above əˈbʌv
abroad əˈbrɔd

Confirmed locale is correct:

$ echo $LANG

en_US.UTF-8

A command to take the second field, split by character, then count how many:

$ cat sample | awk '{print $2}' | grep -o . | sort | uniq -c | sort -n

  1 a
  1 æ
  1 i
  1 v
  2 d
  2 t
  3 e
  3 l
  3 ɔ
  3 r
  4 n
  9 b
 11 ə
 17 ɪ

Where is ʃ and ˈ? They aren't combining characters or anything special. Note that other UTF-8 characters are pulled out: ɔ, ə and ɪ, for example.

BTW using sed 's/\(.\)/\1\n/g' has nearly the same results as grep -o ., except it adds a line for '\n'.

Is there something I'm missing? Does grep have a hidden UTF-8 option?

In case it matters I'm using Ubuntu 12.04.2 LTS.

share|improve this question

2 Answers 2

up vote 3 down vote accepted

The problem is that sort and uniq are using collation information for the locale. Switching the locale off for the two commands works:

cat sample | awk '{print $2}' | grep -o . | LC_ALL=C sort | LC_ALL=C uniq -c | sort -n
      1 ʊ
      1 ʌ
      1 a
      1 æ
      1 i
      1 v
      2 ʃ
      2 d
      2 t
      3 e
      3 l
      3 ɔ
      3 r
      4 ɪ
      4 n
      9 ˈ
      9 b
     11 ə
share|improve this answer
    
While I agree switch locales off fixes the problem. I'm not sure about your interpretation. I can't see why printf '%s\n' ɪ ʃ | uniq -c should return 2 ɪ in any locale. It looks more like a bug in GNU uniq to me. Would you care to expand? –  Stéphane Chazelas Aug 9 '13 at 15:29
    
@StephaneChazelas: I am not sure but I guess the two characters are "equivalent" under the locale (they correspond to nothing). –  choroba Aug 9 '13 at 15:30
2  
OK, I suppose sort can't guarantee that identical (as in byte-to-byte) lines are adjacent, so uniq has to use strcoll there. So your answer is perfectly to the point. And sadly, we have to set LC_ALL to C for both uniq and sort whenever we need to use uniq. –  Stéphane Chazelas Aug 9 '13 at 18:16
1  
@StephaneChazelas: Nice analysis. I agree that the result is unfortunate, but inevitable. –  choroba Aug 9 '13 at 18:19
2  
@derobert, yes U0234 to U07FF all sort the same (glibc 2.17). That can't be right. –  Stéphane Chazelas Aug 9 '13 at 18:29

by using 'C' locale however you lose human collating (like making 'a' and 'A' equivalent).

if you need both to collate and to handle some chars unhandled by glibc locale data; you can create your own locale by expanding default collation.

You can copy the definition of your current locale (eg /usr/share/i18n/locales/en_US ) to another name. Then edit it, and in the LC_COLLATE section have:

LC_COLLATE
copy "iso14651_t1"

reorder-after <e>
<U0259> <e>;<PCL>;<MIN>;IGNORE
reorder-after <s>
<U0283> <s>;<PCL>;<MIN>;IGNORE
reorder-end

END LC_COLLATE

compile it with: localedef -f ./yourmodifiedfile -t UTF-8 ./someplace then you can use LC_ALL=./someplace instead of LC_ALL=C

if you want to use that regularly, put the created directory with the other standard locales (usually /usr/share/locale or /usr/lib/locale ) and name it in a standard way (eg, if it is based on en_US you could name it "en_US@IPA" for example. Then you can set up your locales to have LC_COLLATE=en_US@IPA permanently (note you must not define LC_ALL if you want to individually define some LC_* variables)

Note also U+02C8 is a modifier, and so should rightfully be ignored in collation. But if you need to handle it as a separate character, you can use it instead (ascii single quote put as same (for collate view) as U+02C8, as that is how it is often typed):

# defines a handy symbol, to group together similar chars
collating-symbol <'>

# define 
reorder-after <z>
<'>

reorder-after <e>
<U0259> <e>;<PCL>;<MIN>;IGNORE
reorder-after <s>
<U0283> <s>;<PCL>;<MIN>;IGNORE
reorder-after <'>
<U0027> <'>;<BAS>;IGNORE;IGNORE
<U02C8> <'>;<PCL>;IGNORE;IGNORE

reorder-end

lines are: <unicode value> <1st level>;<2d level>;<3d level>;<4th level> the levels are what is used to sort them.

I think (but not tried, I let that as an exercise :) ) that if you just define the last level it will behave as mostly ignored for sorting, but still "different" from the uniq point of view (as long as the chain of all levels is unique the character is unique, I think).

Usually 1st level is a grouping symbol, like for all the e-like letters. 2nd level is usually for the base character, there are several other symbols for various accented versions, and (peculiar?) is used for "special". 3d level is usually used to differentiate uppercase and lower case and things like that.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.