Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Given this file (annotations are not part of file, but form part of explanation)...

x,c,005,b,d,e,y   # nb - dupe of row 4
x,c,007,b,d,e,y   # nb - dupe of row 4 and 5
x,dd,009,b,d,e,y   # nb - dupe of row 6

... I would like to derive the following output:


If column 3 were cut from the file, and then uniq were run over the file, then if the remaining rows had their column three value added back in at the right place, then I'd get the above result.

But I'm really struggling, to come up with something that would do this. I'd welcome an opportunity to learn about linux's text processing utilities.

Performance: Files don't look likely to grow to more than 1MB, and there is only 1 file per day.

Target: Debian GNU/Linux 7 amd64, 256MB / Xeon.

Edit: tweaked example as fields are not fixedwidth, and a solution involving uniq --skip-chars=n will not work as far as I can tell.

share|improve this question
You were on right track looking for the options to uniq - check my updated answer. :) – peterph Sep 17 '13 at 21:57
up vote 17 down vote accepted

With awk, you could do:

awk -F, -vOFS=, '{l=$0; $3=""}; ! ($0 in seen) {print l; seen[$0]}'
share|improve this answer
wow, elegant and simple (and fast, probably, too, using hash lookups to compare with previous line(s)). However, doesn't it also remove duplicates occuring after something in between? (ie, differently from "uniq were run on the file[if 3rd column removed]" as the OP asked? ie: line1="x,a,001,b,c,d,y", then line12="x,a,999,b,c,d,y" would not appear with your solution but (maybe) should?) – Olivier Dulac Sep 17 '13 at 16:25
You're right that it removes lines after something in between, and you're right that uniq wouldn't do that. But if you look at the OP, he seems to have believed that uniq would act the way this script does, so this script is probably what he actually wanted. – The Spooniest Sep 17 '13 at 16:31
@TheSpooniest: good, then definitely +1 to Stephane to read through the XYProblem ^^ – Olivier Dulac Sep 17 '13 at 16:48

The simplest way:

sort -u -t, -k1,2 -k4
  • -u: output only first line of equals
  • -t,: use comma as field separator
  • -k1,2 -k4: sort only on fields 1,2 and 4 and the rest

Another option is rearranging the data with sed (note the GNU option -r) on both sides - this requires the records to be mostly fixed-length, otherwise it's going to fail (and only hardly noticeably):

sed -r       's/^([^,]+,[^,]+)(,[^,]+)(.*)$/\1\3\2/' \
    | sort \
    | uniq -w 12 \
    | sed -r 's/^([^,]+,[^,]+)(.*)(,[^,]+)$/\1\3\2/'

You might want to add another sort at the end to order it by the numbers, if desired (use the -k option to select according to what the sort should be performed - i.e. something like sed -k3 -t,)

In Perl you could for example use the parts on which you want to decide uniqueness as keys in a hash (the values the full lines) and insert into the hash only if the key is not yet defined. This will of course be much more flexible than using sed (or awk), but also more writing (I'm far from a Perl Guru, so it's very likely it can be done in a much more elegant way - see other answers for Perl-like Perl solutions):

use strict;

my %lines;
while (<>) {
    (my $k1, my $v, my $k2) = /^([^,]+,[^,]+,)([^,]+)(,.*)$/;
    my $k = $k1 . $k2;
    if (!exists($lines{$k})) {
        $lines{$k} = $_;

for my $k (sort(keys(%lines))) {
    print $lines{$k};
share|improve this answer
Thanks, unfortunately fields are not fixed width. I've updated the question, apologies. Your example doesn't work on my system for either old or revised test case :( – jon Sep 17 '13 at 10:44
deleting my answer and upvoting yours - seems to roughly implement the algorithm i described. i'd probably use split rather than regexp for field extraction, and it would be a lot simpler with just $lines{$k} = $_ unless $lines{$k}; – cas Sep 17 '13 at 11:15
Brilliant, +1! I was trying to do that with uniq's field options and couldn't, didn't think to use sort's -u. By the way I think that sort -u is a GNU extension, not POSIX, but this will work fine on Linux systems. – terdon Sep 17 '13 at 17:41
@terdon I think you are right that it is an extension. – peterph Sep 17 '13 at 17:45
Nice and elegant solution! (the perl one) However, as a total Perl rookie, it required me a little RTFM to understand what you were doing here. %lines (easily recognizable by its percent sign) is an associative array (aka "hash variable" in Perl lingo), which may accept "real" strings as key identifiers, not just index numbers. This is the element responsible for all that miraculous "magic" done here. – syntaxerror Aug 9 '15 at 11:42

A way to do this with awk | sort | uniq | awk:

awk -F, '{a=$1;$1=$3;$3=a;print}' file | sort -k 2 | uniq -f 1 | awk -v OFS=',' '{a=$1;$1=$3;$3=a;print}'
share|improve this answer

A simpler Perl way would be:

perl -F"," -ane '$a=join(",",@F[0,1,3 .. $#F]); print unless $k{$a}; $k{$a}++' file

The -a splits fields into the @F array and -F"," sets the field delimiter to ,. -n means run the script given by -e on each line of the input file.

The idea is to take an array slice (elements 0,1 and 3 till the end of the array), join them into a string ($a) and use that string as a hash (associative array) reference. You then print the each line only if the hash key has not been seen before.

share|improve this answer
That would say that ab,c,1,d and a,bc,2,d are the same. You need join(",". Also you can optimize by moving the $k{$a}++ into the unless() { } block. And then, that would be equivalent to my awk solution ;-). – Stéphane Chazelas Sep 17 '13 at 17:10
I don't think it would identify ab,c,1,d and a,bc,2,d as identical - comparison is done on reconstructed string (with the commas at the right places). – peterph Sep 17 '13 at 17:25
@peterph yes but that is because I already corrected the error that Stephane spotted and added join(",". – terdon Sep 17 '13 at 17:26
It's just that you don't need $k{$a}++ if $a is already in %k. You could make it shorter with: perl -F, -ane'print if!$k{join",",@F[0,1,3..-1]}++' – Stéphane Chazelas Sep 17 '13 at 19:25
Stephane, your last suggestion does not provide the expected output, terdon's perl in the edited answer does. – bbaassssiiee Sep 18 '13 at 6:49

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.