Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

How can you find unique lines or duplicate lines in a file based on the content of the first field, or the first n characters ?

The uniq command on OS X lacks the -w switch that is available under Linux: -w, --check-chars=N : compare no more than N characters in lines, which solves the problem.

Can it be done without reimplementing uniq in awk ?

The number of fields is not constant. I would like to use only default tools installed on OS X. I use Macports a lot, but in this case, I don't want to use them.

share|improve this question
up vote 5 down vote accepted

How about using sort -u? This can at least sort by the first field:

 sort -k 1,1 -u input_file
share|improve this answer
I used this, and defered the treatement of unique or duplicate lines a bit later with a: while read firstField otherFields; do stuffs.. – alecail Oct 1 '12 at 10:31

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.