5

I'm trying to find duplicates in files, by using:

sort myfile | uniq -d

I noticed that uniq seems to dislike Japanese characters for some reason. For example, if I have a file:

あい
いあ

Then

sort myfile | uniq -d 

Prints

あい

Why is this? Some kind of locale problem?

Edit: this question was marked as a duplicate. While the underlying problem (strcoll) is the same, this question is fundamentally different. Also, the accepted answer to that question isn't the same as the answer to this question, which is to change locale to C.

8
  • What is your locale setting, please?
    – roaima
    Jun 28, 2017 at 7:15
  • In short: yes, it's a locale issue; no, it's not a misconfiguration from you, it's a choice made in the collation definitions of the locale you're using. Jun 28, 2017 at 7:18
  • 1
    LC_COLLATE=C sort myfile | LC_COLLATE=C uniq -d should work, however. I can reproduce this with glibc & GNU sort/uniq. Jun 28, 2017 at 7:20
  • 2
    Yes, you all are correct. It was a locale problem, my locale was LC_ALL=ja_JP.utf8 and changing it temporarily to LC_ALL=C worked.
    – Agargara
    Jun 28, 2017 at 7:32
  • 1
    @Agargara Please add this as an answer for future readers.
    – dr_
    Jun 28, 2017 at 9:33

1 Answer 1

3

Yes, if the locale is en_US.utf8 (as one example), both strings seem equal:

$ printf "%s\n" "いあ" "あい" "いあ" "あい"
いあ
あい
いあ
あい

$ LC_COLLATE=en_US.utf8 bash -c '
    printf "%s\n" "いあ" "あい" "いあ" "あい" |
    sort | 
    uniq '
いあ

If, however, the language is changed to ja_JP, all seems to work correctly:

$ LC_COLLATE=ja_JP.utf8 bash -c '
    printf "%s\n" "いあ" "あい" "いあ" "あい" | 
    sort | 
    uniq '
あい
いあ

It is interesting to note that (in this case) a C locale also work:

$ LC_COLLATE=C bash -c '
    printf "%s\n" "いあ" "あい" "いあ" "あい" |
    sort |
    uniq '
あい
いあ

That only goes to show that en_US is missing the collate order for some code points.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.