1

I want to know the difference between sort -u and uniq. I tried below command to get the count for the unique strings in a file , sort -u and uniq provides the same output from my knowledge. Then why did it provide two different count.

cat test.txt | sort -u | wc -l 
351
cat test.txt | uniq | wc -l
370

Why does it shows two different line count ?

2
  • 2
    Uniq needs sorted input. Is your input sorted?
    – muru
    Commented Aug 25, 2019 at 12:45
  • no my input is not sorted Commented Aug 25, 2019 at 12:45

2 Answers 2

4

Strictly speaking, uniq doesn't need sorted input - but it is true that uniq will only remove consecutive duplicate lines.

The difference is that:

  • sort sorts a file and (using its -u option) can also eliminate duplicate lines, which will now all be consecutive as they have been sorted.

  • uniq deletes consecutive duplicate lines. It also has options to output only duplicated lines (one of each with -d, or all dupes with -D), and can output a count of how many times a line appeared in the input (-c). It also has other options to control how dupes are detected, such as skipping fields and comparing only N characters in a line.

  • the output of sort can, of course, be piped into uniq if you want to combine the features of both.

sort -u is useful when you either don't care about preserving the order of the input file, or actually want it sorted - but want no duplicates at all in the output.

uniq is useful when preserving the input order is required, or when you only care about consecutive dupes - e.g. when you want to uniq an already-sorted file (no need to waste CPU and IOPS sorting it again); or remove all but one of consecutive line feeds between paragraphs (for example, pdftotext often produces lots of blank space between lines or paragraphs - uniq can remove the extras making it easier to read or edit).

4

uniq needs the input to be already sorted. Duplicates that don't appear sequentially won't be eliminated:

~ printf %s\\n 1 2 1 | uniq
1
2
1
~ printf %s\\n 1 2 1 | sort | uniq
1
2
~ printf %s\\n 1 2 1 | sort -u
1
2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .