Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Everywhere I see someone needing to get a sorted, unique list, they always pipe to sort | uniq. I've never seen any examples where someone uses sort -u instead. Why not? What's the difference, and why is it better to use uniq than the unique flag to sort?

share|improve this question
    

4 Answers 4

up vote 36 down vote accepted

sort | uniq existed before sort -u, and is compatible with a wider range of systems, although almost all modern systems do support -u -- it's POSIX. It's mostly a throwback to the days when sort -u didn't exist (and people don't tend to change their methods if the way that they know continues to work, just look at ifconfig vs. ip adoption).

The two were likely merged because removing duplicates within a file requires sorting (at least, in the standard case), and is an extremely common use case of sort. It is also faster internally as a result of being able to do both operations at the same time (and due to the fact that it doesn't require IPC between uniq and sort). Especially if the file is big, sort -u will likely use fewer intermediate files to sort the data.

On my system I consistently get results like this:

$ dd if=/dev/urandom of=/dev/shm/file bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 8.95208 s, 11.7 MB/s
$ time sort -u /dev/shm/file >/dev/null

real        0m0.500s
user        0m0.767s
sys         0m0.167s
$ time sort /dev/shm/file | uniq >/dev/null

real        0m0.772s
user        0m1.137s
sys         0m0.273s

It also doesn't mask the return code of sort, which may be important (in modern shells there are ways to get this, for example, bash's $PIPESTATUS array, but this wasn't always true).

share|improve this answer
8  
I tend to use sort | uniq because 9 times out of 10, I'm actually piping to uniq -c. –  Plutor May 16 '13 at 14:16
    
Note that sort -u was part of 7th Edition UNIX, circa 1979. Versions of sort without support for -u are truly archaic — or were written without attention to the de facto standard before POSIX's de jure standard. See also Stack Overflow Sort & uniq in Linux shell from 2010. –  Jonathan Leffler Feb 18 at 16:34

One difference is that uniq has a number of useful additional options, such as skipping fields for comparison and counting the number of repetitions of a value. sort's -u flag only implements the functionality of the unadorned uniq command.

share|improve this answer
    
Hi and Welcome to Unix & Linux! Please read the How to Answer a Question Guide. This site is a Q&A site not a forum. –  slm May 16 '13 at 13:53
2  
+0.49 for a useful answer, but I would phrase it something like "The output of sort -u can't be passed to uniq to use some of the latter's useful options, such as skipping fields for comparison and counting the number of repetitions." –  l0b0 May 16 '13 at 14:10
7  
+1 to offset the naysayers because "there's no way to do this directly from sort" does answer the question... –  Izkata May 16 '13 at 15:28

With POSIX compliant sorts and uniqs (GNU uniq is currently not compliant in that regard), there's a difference in that sort uses the locale's collating algorithm to compare strings (will typically use strcoll() to compare strings) while uniq checks for byte-value identity (will typically use strcmp()).

That matters for at least two reasons.

  • In some locales, especially on GNU systems, there are different characters that sort the same. For instance, in the en_US.UTF-8 locale on a GNU system, all the ①②③④⑤⑥⑦⑧⑨⑩... characters and many others sort the same because their sort order is not defined. The 0123456789 arabic digits sort the same as their Eastern Arabic Indic counterparts (٠١٢٣٤٥٦٧٨٩).

    For sort -u, ① sorts the same as ② and 0123 the same as ٠١٢٣ so sort -u would retain only one of each, while for uniq (not GNU uniq which uses strcoll() (except with -i)), ① is different from ② and 0123 different from ٠١٢٣, so uniq would consider all 4 unique.

  • strcoll can only compare strings of valid characters (the behaviour is undefined as per POSIX when the input has sequences of bytes that don't form valid characters) while strcmp() doesn't care about characters since it only does byte-to-byte comparison. So that's another reason why sort -u may not give you all the unique lines if some of them don't form valid text. sort|uniq, while still unspecified on non-text input, in practice is more likely to give you unique lines for that reason.

Beside those subtleties, one thing that hasn't been noted so far is that uniq compares whole line lexically, while sort's -u compares based on the sort specification given on the command line.

$ printf '%s\n' 'a b' 'a c' | sort -uk1,1
a b
$ printf '%s\n' 'a b' 'a c' | sort -k1,1 | uniq
a b
a c

$ printf '%s\n' 0 -0 +0 00 '' | sort -n | uniq
0
-0
+0
00

$ printf '%s\n' 0 -0 +0 00 '' | sort -nu
0
share|improve this answer
$ cat need2sort
pma2005
PMA2005
pma2005

$ LC_CTYPE="C" sort -fu need2sort
pma2005

$ LC_CTYPE="C" sort -f need2sort | uniq
PMA2005
pma2005
share|improve this answer
4  
Welcome to U & L SE. Please consider expanding your answer describing what each command does. –  Ramesh Nov 19 '14 at 3:31

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.