removing redundancy from output columns

Question

What is a *NIX way of removing redundancy in a case where I have pairwise comparisons like these in two columns

    A B
    B A
    A C
    A D
    C A
    D A 
    B C
    C B

A B and B A represent the same comparison and I would like to remove such redundancy from the dataset. The final result should be

A B
A C
A D
B C

Answer 1

doit () 
{ 
    awk '{
           key=$1<=$2? $1 FS $2 : $2 FS $1; 
           if (!seen[key]) print $1,$2
           seen[key]=1
    }'
}

$ doit <test
A B
A C
A D
B C
$

(or, getting terser with it 'cause Chris Down's answer's so sweet)

awk '!seen[$1<=$2? $1 FS $2: $2 FS $1]++ {print $1,$2}'

which could be further reduced if you don't care about the spaces in your data

awk '!seen[$1<=$2? $1 FS $2: $2 FS $1]++'

)

The FS is awk's "field separator" variable, used here to guarantee the boundaries between key fields will be properly identified. My original had them run together, $1$2, which as Stephane Chazelas pointed out would have treated A BC and AB C as duplicates.

Answer 2

In Perl:

perl -lane 'print if !$seen{join(" ", sort @F)}++'

This works by sorting and joining the fields (so "C A" would become key "A C"), and adding them to $seen. It will only print the first occurrence, because the conditional clause will only be true when it evaluates to zero (which will only happen the first time this comparison is encountered, due to the postincrement).

Answer 3

If you don't care to preserve the order of elements on unique line, then you can sort each line, then sort the lines and remove duplicates.

awk '{ if ($2 < $1) print $2, $1; else print $1, $2; }' | sort -u