4

I have a CSV file containing 220k names that I want to batch import into a graph database. The batch importer imposes the restriction that each node must occur only once so I want to remove duplicates using sort and uniq like this:

sort all-authors.csv | uniq > unique-authors.csv

Upon investigating missing nodes I observed this odd behavior:

$ grep 이선동 unique-authors.csv
$ grep 이선동 all-authors.csv 
"이선동"
$

The name is passed on by sort but then lost by uniq:

$ sort all-authors.csv | grep 이선동
"이선동"
$ sort all-authors.csv | uniq | grep 이선동
$

It works with other non-English names, e.g.:

$ grep '잔쎈 우베' *csv
all-authors.csv:"잔쎈 우베"
unique-authors.csv:"잔쎈 우베"
$

What is happening here?

edit: added output of locale

 $ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8
$

edit: added output of attempt with single-line file and sort -u

answering pLumo's comment regarding a file only containing the affected string:

$ grep 이선동 all-authors.csv > single-author.csv
$ sort single-author.csv | uniq
"이선동"
$

answering rush's comment regarding sort -u:

$ sort -u all-authors.csv | grep 이선동

I do not have any aliases set for uniq, I verified using alias | grep uniq, which returns nothing.

0

Based on the answer from: https://stackoverflow.com/questions/5982531/behaviour-of-gnu-sort-command-with-non-letter-ascii-characters-such-as-dot-or

This did the trick for me:

Without LC_COLLATE=C

sort test | uniq
이선동

With LC_COLLATE=C

LC_COLLATE=C sort test | LC_COLLATE=C uniq
동선이
동이선
선동이
선이동
이동선
이선동
  • Unfortunately this doesn't work for me, LC_COLLATE=C sort all-authors.csv | LC_COLLATE=C uniq | LC_COLLATE=C grep 이선동 returns nothing. – oschlueter Jun 19 at 21:16
  • Strange, works for me, even with the grep: LC_COLLATE=C sort test | LC_COLLATE=C uniq | grep 이선동 returns: 이선동 – Tamas H. Jun 20 at 5:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.