2

I have a text file that contain multiple lines. Some of the lines have contain same value. What I want is to make sure that all of the lines do not have duplicates or "triplicates".

Sample:

2014-11-21
2013-10-19
2013-06-09
2014-11-21
2014-11-21
2015-02-19
2012-08-12
2015-07-22
2013-06-09

Output:

 2014-11-21
 2013-10-19
 2013-06-09
 2015-02-19
 2012-08-12
 2015-07-22

I tried uniq but it completely removes all the lines with duplicates. What I wanted is retain one.

  • uniq specifically needs its input sorted (so duplicates are on subsequent lines), so if you sort file | uniq you will get all duplicates removed (but the order sorted). – Drav Sloan Aug 4 '15 at 19:19
5

With sort (it will also sort the output):

sort -u file

With awk (it doesn't sort, just filter out the duplicates, like the desired output):

awk '!a[$0]++' file

With perl (same principle as the awk one):

perl -ne 'print unless $a{$_}++' file
  • 1
    With grep -n '' <file | sort -t: -uk2 | sort -t: -nk1,1 | cut -d: -f2- the original order can be restored, as well. – mikeserv Aug 4 '15 at 20:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.