# Difference between cat file.txt | sort -u and cat file.txt | uniq

I want to know the difference between sort -u and uniq. I tried below command to get the count for the unique strings in a file , sort -u and uniq provides the same output from my knowledge. Then why did it provide two different count.

cat test.txt | sort -u | wc -l
351
cat test.txt | uniq | wc -l
370


Why does it shows two different line count ?

• Uniq needs sorted input. Is your input sorted? – muru Aug 25 at 12:45
• no my input is not sorted – Gabrial Johnas Aug 25 at 12:45

uniq needs the input to be already sorted. Duplicates that don't appear sequentially won't be eliminated:

~ printf %s\\n 1 2 1 | uniq
1
2
1
~ printf %s\\n 1 2 1 | sort | uniq
1
2
~ printf %s\\n 1 2 1 | sort -u
1
2


Strictly speaking, uniq doesn't need sorted input - but it is true that uniq will only remove consecutive duplicate lines.

The difference is that:

• sort sorts a file and (using its -u option) can also eliminate duplicate lines, which will now all be consecutive as they have been sorted.

• uniq deletes consecutive duplicate lines. It also has options to output only duplicated lines (one of each with -d, or all dupes with -D), and can output a count of how many times a line appeared in the input (-c). It also has other options to control how dupes are detected, such as skipping fields and comparing only N characters in a line.

• the output of sort can, of course, be piped into uniq if you want to combine the features of both.

sort -u is useful when you either don't care about preserving the order of the input file, or actually want it sorted - but want no duplicates at all in the output.

uniq is useful when preserving the input order is required, or when you only care about consecutive dupes - e.g. when you want to uniq an already-sorted file (no need to waste CPU and IOPS sorting it again); or remove all but one of consecutive line feeds between paragraphs (for example, pdftotext often produces lots of blank space between lines or paragraphs - uniq can remove the extras making it easier to read or edit).