Some UTF-8 characters not being recognized by grep or sed

Question

Trying to determine all characters in a file.

The file sample consists of:

a eɪ
abandon əˈbændən
ability əˈbɪləti
able ˈeɪbəl
able ˈeɪbl
abortion əˈbɔrʃən
abortion əˈbɔrʃn
about əˈbaʊt
above əˈbʌv
abroad əˈbrɔd

Confirmed locale is correct:

$ echo $LANG

en_US.UTF-8

A command to take the second field, split by character, then count how many:

$ cat sample | awk '{print $2}' | grep -o . | sort | uniq -c | sort -n

  1 a
  1 æ
  1 i
  1 v
  2 d
  2 t
  3 e
  3 l
  3 ɔ
  3 r
  4 n
  9 b
 11 ə
 17 ɪ

Where is ʃ and ˈ? They aren't combining characters or anything special. Note that other UTF-8 characters are pulled out: ɔ, ə and ɪ, for example.

BTW using sed 's/\(.\)/\1\n/g' has nearly the same results as grep -o ., except it adds a line for '\n'.

Is there something I'm missing? Does grep have a hidden UTF-8 option?

In case it matters I'm using Ubuntu 12.04.2 LTS.

Answer 1

The problem is that sort and uniq are using collation information for the locale. Switching the locale off for the two commands works:

cat sample | awk '{print $2}' | grep -o . | LC_ALL=C sort | LC_ALL=C uniq -c | sort -n
      1 ʊ
      1 ʌ
      1 a
      1 æ
      1 i
      1 v
      2 ʃ
      2 d
      2 t
      3 e
      3 l
      3 ɔ
      3 r
      4 ɪ
      4 n
      9 ˈ
      9 b
     11 ə

Answer 2

by using 'C' locale however you lose human collating (like making 'a' and 'A' equivalent).

if you need both to collate and to handle some chars unhandled by glibc locale data; you can create your own locale by expanding default collation.

You can copy the definition of your current locale (eg /usr/share/i18n/locales/en_US ) to another name. Then edit it, and in the LC_COLLATE section have:

LC_COLLATE
copy "iso14651_t1"

reorder-after <e>
<U0259> <e>;<PCL>;<MIN>;IGNORE
reorder-after <s>
<U0283> <s>;<PCL>;<MIN>;IGNORE
reorder-end

END LC_COLLATE

compile it with: localedef -f ./yourmodifiedfile -t UTF-8 ./someplace then you can use LC_ALL=./someplace instead of LC_ALL=C

if you want to use that regularly, put the created directory with the other standard locales (usually /usr/share/locale or /usr/lib/locale ) and name it in a standard way (eg, if it is based on en_US you could name it "en_US@IPA" for example. Then you can set up your locales to have LC_COLLATE=en_US@IPA permanently (note you must not define LC_ALL if you want to individually define some LC_* variables)

Note also U+02C8 is a modifier, and so should rightfully be ignored in collation. But if you need to handle it as a separate character, you can use it instead (ascii single quote put as same (for collate view) as U+02C8, as that is how it is often typed):

# defines a handy symbol, to group together similar chars
collating-symbol <'>

# define 
reorder-after <z>
<'>

reorder-after <e>
<U0259> <e>;<PCL>;<MIN>;IGNORE
reorder-after <s>
<U0283> <s>;<PCL>;<MIN>;IGNORE
reorder-after <'>
<U0027> <'>;<BAS>;IGNORE;IGNORE
<U02C8> <'>;<PCL>;IGNORE;IGNORE

reorder-end

lines are: <unicode value> <1st level>;<2d level>;<3d level>;<4th level> the levels are what is used to sort them.

I think (but not tried, I let that as an exercise :) ) that if you just define the last level it will behave as mostly ignored for sorting, but still "different" from the uniq point of view (as long as the chain of all levels is unique the character is unique, I think).

Usually 1st level is a grouping symbol, like for all the e-like letters. 2nd level is usually for the base character, there are several other symbols for various accented versions, and (peculiar?) is used for "special". 3d level is usually used to differentiate uppercase and lower case and things like that.