5

What is a *NIX way of removing redundancy in a case where I have pairwise comparisons like these in two columns

    A B
    B A
    A C
    A D
    C A
    D A 
    B C
    C B

A B and B A represent the same comparison and I would like to remove such redundancy from the dataset. The final result should be

A B
A C
A D
B C
7
doit () 
{ 
    awk '{
           key=$1<=$2? $1 FS $2 : $2 FS $1; 
           if (!seen[key]) print $1,$2
           seen[key]=1
    }'
}
$ doit <test
A B
A C
A D
B C
$

(or, getting terser with it 'cause Chris Down's answer's so sweet)

awk '!seen[$1<=$2? $1 FS $2: $2 FS $1]++ {print $1,$2}'

which could be further reduced if you don't care about the spaces in your data

awk '!seen[$1<=$2? $1 FS $2: $2 FS $1]++'

)

The FS is awk's "field separator" variable, used here to guarantee the boundaries between key fields will be properly identified. My original had them run together, $1$2, which as Stephane Chazelas pointed out would have treated A BC and AB C as duplicates.

5
  • Nice. Do you want to add an explanantion of what is going on in there for those of use who are wanting to learn more awk?
    – jasonwryan
    Nov 18 '13 at 17:28
  • @jasonwryan Adding a note about the FS usage now, is there anything else?
    – jthill
    Nov 18 '13 at 17:38
  • you gotta love awk... usually provides clear and concise answer to everyday problems. Nice one, jthill. I prefer the top version, as it's much clearer (to me, anyway) than the terser ones, at first glance Nov 18 '13 at 20:00
  • @OlivierDulac Thanks. Yeah, perl has better reach, it's not even a contest, but awk is so easily accessible to non-career-IT guys it's my go-to.
    – jthill
    Nov 18 '13 at 20:11
  • 1
    @jthill: rejoice :) swtch.com/~rsc/regexp/regexp1.html (but it could have changed since then... if not, awk and grep 's regexp could be a very good thing compared to Perl's ones... But I guess there is packages (or new versions?) with the same approach in perl?). Off topic here, but I just found out about this and it made awk shine again ^^ Nov 18 '13 at 20:19
5

In Perl:

perl -lane 'print if !$seen{join(" ", sort @F)}++'

This works by sorting and joining the fields (so "C A" would become key "A C"), and adding them to $seen. It will only print the first occurrence, because the conditional clause will only be true when it evaluates to zero (which will only happen the first time this comparison is encountered, due to the postincrement).

0
2

If you don't care to preserve the order of elements on unique line, then you can sort each line, then sort the lines and remove duplicates.

awk '{ if ($2 < $1) print $2, $1; else print $1, $2; }' | sort -u

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.