5

I'm trying to find duplicates in files, by using:

sort myfile | uniq -d

I noticed that uniq seems to dislike Japanese characters for some reason. For example, if I have a file:

あい
いあ

Then

sort myfile | uniq -d 

Prints

あい

Why is this? Some kind of locale problem?

Edit: this question was marked as a duplicate. While the underlying problem (strcoll) is the same, this question is fundamentally different. Also, the accepted answer to that question isn't the same as the answer to this question, which is to change locale to C.

  • What is your locale setting, please? – roaima Jun 28 '17 at 7:15
  • In short: yes, it's a locale issue; no, it's not a misconfiguration from you, it's a choice made in the collation definitions of the locale you're using. – Michael Homer Jun 28 '17 at 7:18
  • 1
    LC_COLLATE=C sort myfile | LC_COLLATE=C uniq -d should work, however. I can reproduce this with glibc & GNU sort/uniq. – Michael Homer Jun 28 '17 at 7:20
  • 2
    Yes, you all are correct. It was a locale problem, my locale was LC_ALL=ja_JP.utf8 and changing it temporarily to LC_ALL=C worked. – Agargara Jun 28 '17 at 7:32
  • 1
    @Agargara Please add this as an answer for future readers. – dr01 Jun 28 '17 at 9:33
2

Yes, if the locale is en_US.utf8 (as one example), both strings seem equal:

$ printf "%s\n" "いあ" "あい" "いあ" "あい"
いあ
あい
いあ
あい

$ LC_COLLATE=en_US.utf8 bash -c '
    printf "%s\n" "いあ" "あい" "いあ" "あい" |
    sort | 
    uniq '
いあ

If, however, the language is changed to ja_JP, all seems to work correctly:

$ LC_COLLATE=ja_JP.utf8 bash -c '
    printf "%s\n" "いあ" "あい" "いあ" "あい" | 
    sort | 
    uniq '
あい
いあ

It is interesting to note that (in this case) a C locale also work:

$ LC_COLLATE=C bash -c '
    printf "%s\n" "いあ" "あい" "いあ" "あい" |
    sort |
    uniq '
あい
いあ

That only goes to show that en_US is missing the collate order for some code points.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.