62
$ cat data.txt 
aaaaaa
aaaaaa
cccccc
aaaaaa
aaaaaa
bbbbbb
$ cat data.txt | uniq
aaaaaa
cccccc
aaaaaa
bbbbbb
$ cat data.txt | sort | uniq
aaaaaa
bbbbbb
cccccc
$

The result that I need is to display all the lines from the original file removing all the duplicates (not just the consecutive ones), while maintaining the original order of statements in the file.

Here, in this example, the result that I actually was looking for was

aaaaaa
cccccc
bbbbbb

How can I perform this generalized uniq operation in general?

0

5 Answers 5

76
perl -ne 'print unless $seen{$_}++' data.txt

Or, if you must have a useless use of cat:

cat data.txt | perl -ne 'print unless $seen{$_}++'

Here's an awk translation, for systems that lack Perl:

awk '!seen[$0]++' data.txt
cat data.txt | awk '!seen[$0]++'
13
  • 4
    A slightly shorter awk script is { if (!seen[$0]++) print }
    – camh
    Commented Apr 24, 2011 at 22:32
  • 1
    general info: I time tested both perl and awk... awk was faster by 25+%)... 450000 unique lines (doubled up) to make 900000 lines (4 bytes per line)
    – Peter.O
    Commented Apr 25, 2011 at 0:48
  • 1
    @fred, unless your file is truly huge, either version takes longer to type in than it does to run.
    – cjm
    Commented Apr 25, 2011 at 3:32
  • 15
    The awk version can be made even shorter by leaving out the if, print, parentheses, and braces: awk '!seen[$0]++' Commented Apr 25, 2011 at 6:29
  • 3
    @Legate, it's the name of an array in which we're recording every line we've seen. You could change it to '!LarryWall[$0]++' for all awk cares, but "seen" helps people understand the program better.
    – cjm
    Commented Jan 17, 2012 at 19:14
21

john has a tool called unique:

usr@srv % cat data.txt | unique out
usr@srv % cat out
aaaaaa
cccccc
bbbbbb

To achieve the same without additional tools in a single commandline is a bit more complex:

usr@srv % cat data.txt | nl | sort -k 2 | uniq -f 1 | sort -n | sed 's/\s*[0-9]\+\s\+//'
aaaaaa
cccccc
bbbbbb

nl prints line numbers in front of the lines, so if we sort/uniq behind them, we can restore the original order of the lines. sed just deletes the line numbers afterwards ;)

2
  • is there any combination of common linux commands that could do the same?
    – Lazer
    Commented Apr 24, 2011 at 20:45
  • @Totor - see menkus' reply to a similar comment . @binfalse - your second solution does not work (maybe it works with this trivial sample but it doesn't work with some real life input). Please fix that, e.g. this should always work: nl -ba -nrz data.txt | sort -k2 -u | sort | cut -f2 Commented Nov 7, 2015 at 19:33
12

I prefer to use this:

cat -n data.txt | sort --key=2.1 -b -u | sort -n | cut -c8-

cat -n adds line numbers,

sort --key=2.1 -b -u sorts on the second field (after the added line numbers), ignoring leading blanks, keeping unique lines

sort -n sorts in strict numeric order

cut -c8- keep all characters from column 8 to EOL (i.e., omit the line numbers we included)

3
  • 7
    >How to get only the unique results without having to sort data? >without having to sort data Commented Jul 30, 2013 at 6:40
  • 8
    'without having to sort the data' only appears in the title. The actual need is to: "display all the lines from the original file removing all the duplicates (not just the consecutive ones), while maintaining the original order of statements in the file."
    – menkus
    Commented Aug 7, 2013 at 18:00
  • The same thing, but using tabs as the delimiter for the line numbers: awk -v OFS='\t' '{ print NR, $0 }' data.txt | sort -u -b -k 2 | sort -n | cut -f 2-
    – Kusalananda
    Commented Jun 29, 2022 at 14:14
3

Perl has a module that you can use that includes a function called uniq. So if you ave your data loaded in an array in Perl you simply call the function like this to make it unique, yet still maintain the original order.

use List::MoreUtils qw(uniq)    
@output = uniq(@output);

You can read more about this module here: List::MoreUtils

1
  • Can this handle huge files, e.g. 500 GB?
    – Boy
    Commented Dec 20, 2017 at 19:47
0

Using Raku (formerly known as Perl_6)

~$ raku -e '.put for lines.unique;'  file

OR (more awk-like syntax):

~$ raku -ne 'state %h; .put unless %h{$_}++ ;'  file

Sample Input:

aaaaaa
aaaaaa
cccccc
aaaaaa
aaaaaa
bbbbbb

Sample Output:

aaaaaa
cccccc
bbbbbb

https://docs.raku.org
https://raku.org

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .