Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

What is a *NIX way of removing redundancy in a case where I have pairwise comparisons like these in two columns

    A B
    B A
    A C
    A D
    C A
    D A 
    B C
    C B

A B and B A represent the same comparison and I would like to remove such redundancy from the dataset. The final result should be

share|improve this question
up vote 7 down vote accepted
doit () 
    awk '{
           key=$1<=$2? $1 FS $2 : $2 FS $1; 
           if (!seen[key]) print $1,$2
$ doit <test

(or, getting terser with it 'cause Chris Down's answer's so sweet)

awk '!seen[$1<=$2? $1 FS $2: $2 FS $1]++ {print $1,$2}'

which could be further reduced if you don't care about the spaces in your data

awk '!seen[$1<=$2? $1 FS $2: $2 FS $1]++'


The FS is awk's "field separator" variable, used here to guarantee the boundaries between key fields will be properly identified. My original had them run together, $1$2, which as Stephane Chazelas pointed out would have treated A BC and AB C as duplicates.

share|improve this answer
Nice. Do you want to add an explanantion of what is going on in there for those of use who are wanting to learn more awk? – jasonwryan Nov 18 '13 at 17:28
@jasonwryan Adding a note about the FS usage now, is there anything else? – jthill Nov 18 '13 at 17:38
you gotta love awk... usually provides clear and concise answer to everyday problems. Nice one, jthill. I prefer the top version, as it's much clearer (to me, anyway) than the terser ones, at first glance – Olivier Dulac Nov 18 '13 at 20:00
@OlivierDulac Thanks. Yeah, perl has better reach, it's not even a contest, but awk is so easily accessible to non-career-IT guys it's my go-to. – jthill Nov 18 '13 at 20:11
@jthill: rejoice :) swtch.com/~rsc/regexp/regexp1.html (but it could have changed since then... if not, awk and grep 's regexp could be a very good thing compared to Perl's ones... But I guess there is packages (or new versions?) with the same approach in perl?). Off topic here, but I just found out about this and it made awk shine again ^^ – Olivier Dulac Nov 18 '13 at 20:19

In Perl:

perl -lane 'print if !$seen{join(" ", sort @F)}++'

This works by sorting and joining the fields (so "C A" would become key "A C"), and adding them to $seen. It will only print the first occurrence, because the conditional clause will only be true when it evaluates to zero (which will only happen the first time this comparison is encountered, due to the postincrement).

share|improve this answer

If you don't care to preserve the order of elements on unique line, then you can sort each line, then sort the lines and remove duplicates.

awk '{ if ($2 < $1) print $2, $1; else print $1, $2; }' | sort -u
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.