Is there some better solution for printing unique lines other than a combination of sort and uniq?

  • 1
    What do you mean by "better"? – gabe. Mar 23 '11 at 13:31
  • @gabe Not requiring the entire file to be stored in memory for example. – Let_Me_Be Mar 23 '11 at 13:46

To print each identical line only one, in any order:

sort -u

To print only the unique lines, in any order:

sort | uniq -u

To print each identical line only once, in the order of their first occurrence: (for each line, print the line if it hasn't been seen yet, then in any case increment the seen counter)

awk '!seen[$0] {print}

To print only the unique lines, in the order of their first occurrence: (record each line in seen, and also in lines if it's the first occurrence; at the end of the input, print the lines in order of occurrence but only the ones seen only once)

awk '!seen[$0]++ {lines[i++]=$0}
     END {for (i in lines) if (seen[lines[i]]==1) print lines[i]}'
  • 8
    how about awk '!seen[$0]++ {print}'? – asoundmove Mar 23 '11 at 3:26
  • 9
    Or even shorter awk '!seen[$0]++', since the {print} is implied by an empty command. – quazgar Jun 4 '15 at 10:23

Some (most?) versions of sort have a -u flag that does the uniq part directly. Might be some line length restrictions depending on the implementation though, but you had those already with plain sort|uniq.

  • 1
    Er? sort -u goes back to V7 at least. – geekosaur Mar 22 '11 at 22:46
  • Hum... I thought I remembered Solaris or AIX not having that. I'm wrong though, they both have it. – Mat Mar 22 '11 at 22:50
  • Solaris and AIX have -u but also have a 512-character line length restriction. (Actually, I think somewhere around Solaris 9 Sun upped it to 5120. GNU still wins, though.) – geekosaur Mar 22 '11 at 22:52
  • @geekosaur: are you sure? The work done to remove the 512-byte limit on line length in sort was documented in 'Theory and Practice in the Construction of a Working Sort Routine' by J P Linderman, Bell System Technical. Journal, 63, 1827- 1843 (1984). – Jonathan Leffler Mar 23 '11 at 3:32

Does Perl work for you? It can keep the lines in the original order, even if the duplicates are not adjacent. You could also code it in Python, or awk.

while (<>) {
    print if $lines{$_}++ == 0;

Which can be shortened to just

perl -ne 'print unless $lines{$_}++;'

Given input file:


It yields the output:

  • Where is $lines getting defined? – Gregg Leventhal Jul 20 '14 at 3:09
  • It isn't. Since there isn't a use strict; or use warnings; (actually, it is strict that is most relevant here), there is no complaint about using %lines before it is defined. If run with strictures, there'd need to be a line my %lines; before the loop. Note, too, that the hash is %lines; one element of the hash is referenced using the $lines{$_} notation. – Jonathan Leffler Jul 20 '14 at 4:47
  • I think the sort solutions may be better for large amount of data (the OP was concerned about "storing the entire file in memory"). sort will perform an out-of-core sort if the data is larger than the available memory. – Kusalananda Apr 6 '17 at 8:29

For the last part of the answer mentioned in : Printing unique lines by @Gilles as an answer to this question, I tried to eliminate the need for using two hashes.

This solution is for : To print only the unique lines, in the order of their first occurrence:

awk '{counter[$0]++} END {for (line in counter) if (counter[line]==1) print line}'

Here, "counter" stores a count of each line that is similar to the one processed earlier.
At the end, we print only those lines, that have counter value as 1.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.