Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Is there some better solution for printing unique lines other than a combination of sort and uniq?

share|improve this question
    
What do you mean by "better"? –  gabe. Mar 23 '11 at 13:31
    
@gabe Not requiring the entire file to be stored in memory for example. –  Let_Me_Be Mar 23 '11 at 13:46

3 Answers 3

up vote 8 down vote accepted

To print each identical line only one, in any order:

sort -u

To print only the unique lines, in any order:

sort | uniq -u

To print each identical line only once, in the order of their first occurrence: (for each line, print the line if it hasn't been seen yet, then in any case increment the seen counter)

awk '!seen[$0] {print}
     {++seen[$0]}'

To print only the unique lines, in the order of their first occurrence: (record each line in seen, and also in lines if it's the first occurrence; at the end of the input, print the lines in order of occurrence but only the ones seen only once)

awk '!seen[$0]++ {lines[i++]=$0}
     END {for (i in lines) if (seen[lines[i]]==1) print lines[i]}'
share|improve this answer
3  
how about awk '!seen[$0]++ {print}'? –  asoundmove Mar 23 '11 at 3:26

Some (most?) versions of sort have a -u flag that does the uniq part directly. Might be some line length restrictions depending on the implementation though, but you had those already with plain sort|uniq.

share|improve this answer
1  
Er? sort -u goes back to V7 at least. –  geekosaur Mar 22 '11 at 22:46
    
Hum... I thought I remembered Solaris or AIX not having that. I'm wrong though, they both have it. –  Mat Mar 22 '11 at 22:50
    
Solaris and AIX have -u but also have a 512-character line length restriction. (Actually, I think somewhere around Solaris 9 Sun upped it to 5120. GNU still wins, though.) –  geekosaur Mar 22 '11 at 22:52
    
@geekosaur: are you sure? The work done to remove the 512-byte limit on line length in sort was documented in 'Theory and Practice in the Construction of a Working Sort Routine' by J P Linderman, Bell System Technical. Journal, 63, 1827- 1843 (1984). –  Jonathan Leffler Mar 23 '11 at 3:32

Does Perl work for you? It can keep the lines in the original order, even if the duplicates are not adjacent. You could also code it in Python, or awk.

while (<>)
{
    print if $lines{$_}++ = 0;
}

Given input file:

abc
def
abc
ghi
abc
def
abc
ghi
jkl

It yields the output:

abc
def
ghi
jkl
share|improve this answer
    
Where is $lines getting defined? –  Gregg Leventhal Jul 20 at 3:09
    
It isn't. Since there isn't a use strict; or use warnings; (actually, it is strict that is most relevant here), there is no complaint about using %lines before it is defined. If run with strictures, there'd need to be a line my %lines; before the loop. Note, too, that the hash is %lines; one element of the hash is referenced using the $lines{$_} notation. –  Jonathan Leffler Jul 20 at 4:47

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.