0

I want to know the difference between sort -u and uniq. I tried below command to get the count for the unique strings in a file , sort -u and uniq provides the same output from my knowledge. Then why did it provide two different count.

cat test.txt | sort -u | wc -l 
351
cat test.txt | uniq | wc -l
370

Why does it shows two different line count ?

  • 2
    Uniq needs sorted input. Is your input sorted? – muru Aug 25 at 12:45
  • no my input is not sorted – Gabrial Johnas Aug 25 at 12:45
3

uniq needs the input to be already sorted. Duplicates that don't appear sequentially won't be eliminated:

~ printf %s\\n 1 2 1 | uniq
1
2
1
~ printf %s\\n 1 2 1 | sort | uniq
1
2
~ printf %s\\n 1 2 1 | sort -u
1
2
3

Strictly speaking, uniq doesn't need sorted input - but it is true that uniq will only remove consecutive duplicate lines.

The difference is that:

  • sort sorts a file and (using its -u option) can also eliminate duplicate lines, which will now all be consecutive as they have been sorted.

  • uniq deletes consecutive duplicate lines. It also has options to output only duplicated lines (one of each with -d, or all dupes with -D), and can output a count of how many times a line appeared in the input (-c). It also has other options to control how dupes are detected, such as skipping fields and comparing only N characters in a line.

  • the output of sort can, of course, be piped into uniq if you want to combine the features of both.

sort -u is useful when you either don't care about preserving the order of the input file, or actually want it sorted - but want no duplicates at all in the output.

uniq is useful when preserving the input order is required, or when you only care about consecutive dupes - e.g. when you want to uniq an already-sorted file (no need to waste CPU and IOPS sorting it again); or remove all but one of consecutive line feeds between paragraphs (for example, pdftotext often produces lots of blank space between lines or paragraphs - uniq can remove the extras making it easier to read or edit).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.