59
$ cat data.txt 
aaaaaa
aaaaaa
cccccc
aaaaaa
aaaaaa
bbbbbb
$ cat data.txt | uniq
aaaaaa
cccccc
aaaaaa
bbbbbb
$ cat data.txt | sort | uniq
aaaaaa
bbbbbb
cccccc
$

The result that I need is to display all the lines from the original file removing all the duplicates (not just the consecutive ones), while maintaining the original order of statements in the file.

Here, in this example, the result that I actually was looking for was

aaaaaa
cccccc
bbbbbb

How can I perform this generalized uniq operation in general?

4 Answers 4

70
perl -ne 'print unless $seen{$_}++' data.txt

Or, if you must have a useless use of cat:

cat data.txt | perl -ne 'print unless $seen{$_}++'

Here's an awk translation, for systems that lack Perl:

awk '!seen[$0]++' data.txt
cat data.txt | awk '!seen[$0]++'
13
  • 4
    A slightly shorter awk script is { if (!seen[$0]++) print }
    – camh
    Apr 24, 2011 at 22:32
  • 1
    @fred, unless your file is truly huge, either version takes longer to type in than it does to run.
    – cjm
    Apr 25, 2011 at 3:32
  • 13
    The awk version can be made even shorter by leaving out the if, print, parentheses, and braces: awk '!seen[$0]++' Apr 25, 2011 at 6:29
  • 2
    @Legate, it's the name of an array in which we're recording every line we've seen. You could change it to '!LarryWall[$0]++' for all awk cares, but "seen" helps people understand the program better.
    – cjm
    Jan 17, 2012 at 19:14
  • 1
    @Sadi, that really should have been asked as a question, not a comment. But some of the lines in that file end in a space, and some don't. These commands consider the entire line significant, including the whitespace at the end.
    – cjm
    May 17, 2014 at 17:07
16

john has a tool called unique:

usr@srv % cat data.txt | unique out
usr@srv % cat out
aaaaaa
cccccc
bbbbbb

To achieve the same without additional tools in a single commandline is a bit more complex:

usr@srv % cat data.txt | nl | sort -k 2 | uniq -f 1 | sort -n | sed 's/\s*[0-9]\+\s\+//'
aaaaaa
cccccc
bbbbbb

nl prints line numbers in front of the lines, so if we sort/uniq behind them, we can restore the original order of the lines. sed just deletes the line numbers afterwards ;)

2
  • is there any combination of common linux commands that could do the same?
    – Lazer
    Apr 24, 2011 at 20:45
  • @Totor - see menkus' reply to a similar comment . @binfalse - your second solution does not work (maybe it works with this trivial sample but it doesn't work with some real life input). Please fix that, e.g. this should always work: nl -ba -nrz data.txt | sort -k2 -u | sort | cut -f2 Nov 7, 2015 at 19:33
8

I prefer to use this:

cat -n data.txt | sort --key=2.1 -b -u | sort -n | cut -c8-

cat -n adds line numbers,

sort --key=2.1 -b -u sorts on the second field (after the added line numbers), ignoring leading blanks, keeping unique lines

sort -n sorts in strict numeric order

cut -c8- keep all characters from column 8 to EOL (i.e., omit the line numbers we included)

3
  • 7
    >How to get only the unique results without having to sort data? >without having to sort data Jul 30, 2013 at 6:40
  • 7
    'without having to sort the data' only appears in the title. The actual need is to: "display all the lines from the original file removing all the duplicates (not just the consecutive ones), while maintaining the original order of statements in the file."
    – menkus
    Aug 7, 2013 at 18:00
  • 1
    @menkus the key is "while maintaining the original order of statements in the file". This answer doesn't achieve that. May 6, 2018 at 15:02
2

Perl has a module that you can use that includes a function called uniq. So if you ave your data loaded in an array in Perl you simply call the function like this to make it unique, yet still maintain the original order.

use List::MoreUtils qw(uniq)    
@output = uniq(@output);

You can read more about this module here: List::MoreUtils

1
  • Can this handle huge files, e.g. 500 GB?
    – Boy
    Dec 20, 2017 at 19:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.