I'm trying to combine both the American and British dictionaries into one large dictionaries, and I'm trying to remove all the duplicates from the superset, but it seems that uniq is not outputting words like, "épée" or single letters.

This is what I've tried using:

LC_COLLATE=en_US.UTF-8 cat american-english british-english |sort|uniq -u > unique_sorted_combined_dict

If I just do this:

LC_COLLATE=en_US.UTF-8 cat american-english british-english |sort > sorted_combined_dict

"épée" and other such words do show up, along with single letters.

Is there something I'm missing here with uniq?

I should note that I'm using uniq from the GNU coreutils on Ubuntu 12.10, if that makes any difference.

  • 1
    What if you do 'LC_COLLATE=en_US.UTF-8 cat american-english british-english |sort > temp && cat temp | uniq > sorted_combined_dict' ? – schaiba Feb 4 '13 at 20:57
  • I decided, per your comment, to just take out the -u flag, and now it's working. I don't know why -u is not giving me the expected results though. – supercheetah Feb 4 '13 at 21:06

You're setting LC_COLLATE for the cat command only (which doesn't make use of it), while you need to set it for sort and uniq.

Also, you may need to set LC_CTYPE to something utf-8, otherwise it will cause confusion. I would set LC_ALL to en_US.UTF-8

uniq -u only reports unique lines. So, if those single letter words all appear several times, it's normal that they don't show up.

On my system, épée does appear twice:

$ cat american-english british-english | sort | grep -x 'épée'

Maybe you meant sort | uniq or sort -u.

| improve this answer | |
  • Yeah, I obviously misunderstood what the -u flag does, and I think that was the cause of my problems. Thanks! – supercheetah Feb 5 '13 at 23:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.