Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
$ cat data.txt 
aaaaaa
aaaaaa
cccccc
aaaaaa
aaaaaa
bbbbbb
$ cat data.txt | uniq
aaaaaa
cccccc
aaaaaa
bbbbbb
$ cat data.txt | sort | uniq
aaaaaa
bbbbbb
cccccc
$

The result that I need is to display all the lines from the original file removing all the duplicates (not just the consecutive ones), while maintaining the original order of statements in the file.

Here, in this example, the result that I actually was looking for was

aaaaaa
cccccc
bbbbbb

How can I perform this generalized uniq operation in general?

share|improve this question
add comment

4 Answers 4

up vote 25 down vote accepted
perl -ne 'print unless $seen{$_}++' data.txt

Or, if you must have a useless use of cat:

cat data.txt | perl -ne 'print unless $seen{$_}++'

Here's an awk translation, for systems that lack Perl:

awk '!seen[$0]++' data.txt
cat data.txt | awk '!seen[$0]++'
share|improve this answer
1  
A slightly shorter awk script is { if (!seen[$0]++) print } –  camh Apr 24 '11 at 22:32
1  
@fred, unless your file is truly huge, either version takes longer to type in than it does to run. –  cjm Apr 25 '11 at 3:32
6  
The awk version can be made even shorter by leaving out the if, print, parentheses, and braces: awk '!seen[$0]++' –  Gordon Davisson Apr 25 '11 at 6:29
1  
@Legate, it's the name of an array in which we're recording every line we've seen. You could change it to '!LarryWall[$0]++' for all awk cares, but "seen" helps people understand the program better. –  cjm Jan 17 '12 at 19:14
1  
@Sadi, that really should have been asked as a question, not a comment. But some of the lines in that file end in a space, and some don't. These commands consider the entire line significant, including the whitespace at the end. –  cjm May 17 at 17:07
show 9 more comments

john has a tool called unique:

usr@srv % cat data.txt | unique out
usr@srv % cat out
aaaaaa
cccccc
bbbbbb

To achieve the same without additional tools in a single commandline is a bit more complex:

usr@srv % cat data.txt | nl | sort -k 2 | uniq -f 1 | sort -n | sed 's/\s*[0-9]\+\s\+//'
aaaaaa
cccccc
bbbbbb

nl prints line numbers in front of the lines, so if we sort/uniq behind them, we can restore the original order of the lines. sed just deletes the line numbers afterwards ;)

share|improve this answer
    
is there any combination of common linux commands that could do the same? –  Lazer Apr 24 '11 at 20:45
3  
What did you miss in "without having to sort data"? –  Totor Jul 29 '13 at 19:00
add comment

I prefer to use this:

cat -n data.txt | sort --key=2.1 -b -u | sort -n | cut -c8-

cat -n adds line numbers,

sort --key=2.1 -b -u sorts on the second field (after the added line numbers), ignoring leading blanks, keeping unique lines

sort -n sorts in strict numeric order

cut -c8- keep all characters from column 8 to EOL (i.e., omit the line numbers we included)

share|improve this answer
1  
>How to get only the unique results without having to sort data? >without having to sort data –  Jan Wikholm Jul 30 '13 at 6:40
1  
'without having to sort the data' only appears in the title. The actual need is to: "display all the lines from the original file removing all the duplicates (not just the consecutive ones), while maintaining the original order of statements in the file." –  menkus Aug 7 '13 at 18:00
add comment

Perl has a module that you can use that includes a function called uniq. So if you ave your data loaded in an array in Perl you simply call the function like this to make it unique, yet still maintain the original order.

use List::MoreUtils qw(uniq)    
@output = uniq(@output);

You can read more about this module here: List::MoreUtils

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.