Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Is there some better solution for printing unique lines other than a combination of sort and uniq?

share|improve this question
What do you mean by "better"? – gabe. Mar 23 '11 at 13:31
@gabe Not requiring the entire file to be stored in memory for example. – Let_Me_Be Mar 23 '11 at 13:46
up vote 13 down vote accepted

To print each identical line only one, in any order:

sort -u

To print only the unique lines, in any order:

sort | uniq -u

To print each identical line only once, in the order of their first occurrence: (for each line, print the line if it hasn't been seen yet, then in any case increment the seen counter)

awk '!seen[$0] {print}

To print only the unique lines, in the order of their first occurrence: (record each line in seen, and also in lines if it's the first occurrence; at the end of the input, print the lines in order of occurrence but only the ones seen only once)

awk '!seen[$0]++ {lines[i++]=$0}
     END {for (i in lines) if (seen[lines[i]]==1) print lines[i]}'
share|improve this answer
how about awk '!seen[$0]++ {print}'? – asoundmove Mar 23 '11 at 3:26
Or even shorter awk '!seen[$0]++', since the {print} is implied by an empty command. – quazgar Jun 4 '15 at 10:23

Some (most?) versions of sort have a -u flag that does the uniq part directly. Might be some line length restrictions depending on the implementation though, but you had those already with plain sort|uniq.

share|improve this answer
Er? sort -u goes back to V7 at least. – geekosaur Mar 22 '11 at 22:46
Hum... I thought I remembered Solaris or AIX not having that. I'm wrong though, they both have it. – Mat Mar 22 '11 at 22:50
Solaris and AIX have -u but also have a 512-character line length restriction. (Actually, I think somewhere around Solaris 9 Sun upped it to 5120. GNU still wins, though.) – geekosaur Mar 22 '11 at 22:52
@geekosaur: are you sure? The work done to remove the 512-byte limit on line length in sort was documented in 'Theory and Practice in the Construction of a Working Sort Routine' by J P Linderman, Bell System Technical. Journal, 63, 1827- 1843 (1984). – Jonathan Leffler Mar 23 '11 at 3:32

Does Perl work for you? It can keep the lines in the original order, even if the duplicates are not adjacent. You could also code it in Python, or awk.

while (<>)
    print if $lines{$_}++ = 0;

Given input file:


It yields the output:

share|improve this answer
Where is $lines getting defined? – Gregg Leventhal Jul 20 '14 at 3:09
It isn't. Since there isn't a use strict; or use warnings; (actually, it is strict that is most relevant here), there is no complaint about using %lines before it is defined. If run with strictures, there'd need to be a line my %lines; before the loop. Note, too, that the hash is %lines; one element of the hash is referenced using the $lines{$_} notation. – Jonathan Leffler Jul 20 '14 at 4:47

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.