Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Is there some better solution for printing unique lines other than a combination of sort and uniq?

share|improve this question
    
What do you mean by "better"? –  gabe. Mar 23 '11 at 13:31
    
@gabe Not requiring the entire file to be stored in memory for example. –  Let_Me_Be Mar 23 '11 at 13:46
add comment

3 Answers

up vote 8 down vote accepted

To print each identical line only one, in any order:

sort -u

To print only the unique lines, in any order:

sort | uniq -u

To print each identical line only once, in the order of their first occurrence: (for each line, print the line if it hasn't been seen yet, then in any case increment the seen counter)

awk '!seen[$0] {print}
     {++seen[$0]}'

To print only the unique lines, in the order of their first occurrence: (record each line in seen, and also in lines if it's the first occurrence; at the end of the input, print the lines in order of occurrence but only the ones seen only once)

awk '!seen[$0]++ {lines[i++]=$0}
     END {for (i in lines) if (seen[lines[i]]==1) print lines[i]}'
share|improve this answer
3  
how about awk '!seen[$0]++ {print}'? –  asoundmove Mar 23 '11 at 3:26
add comment

Some (most?) versions of sort have a -u flag that does the uniq part directly. Might be some line length restrictions depending on the implementation though, but you had those already with plain sort|uniq.

share|improve this answer
1  
Er? sort -u goes back to V7 at least. –  geekosaur Mar 22 '11 at 22:46
    
Hum... I thought I remembered Solaris or AIX not having that. I'm wrong though, they both have it. –  Mat Mar 22 '11 at 22:50
    
Solaris and AIX have -u but also have a 512-character line length restriction. (Actually, I think somewhere around Solaris 9 Sun upped it to 5120. GNU still wins, though.) –  geekosaur Mar 22 '11 at 22:52
    
@geekosaur: are you sure? The work done to remove the 512-byte limit on line length in sort was documented in 'Theory and Practice in the Construction of a Working Sort Routine' by J P Linderman, Bell System Technical. Journal, 63, 1827- 1843 (1984). –  Jonathan Leffler Mar 23 '11 at 3:32
add comment

Does Perl work for you? It can keep the lines in the original order, even if the duplicates are not adjacent. You could also code it in Python, or awk.

while (<>)
{
    print if $lines{$_}++ = 0;
}

Given input file:

abc
def
abc
ghi
abc
def
abc
ghi
jkl

It yields the output:

abc
def
ghi
jkl
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.