0

I have previously asked a question about keeping the first instance of duplicates and the solution provided has been working perfectly until now (see Keeping first instance of duplicates).

However, I now have the scenario where the values in Columns C and D could be the same but in different orders and I only want to keep one (either order will do).

Example input:

A B C D E F G
1 2 T TACA 3 2 Q
9 3 A C 9 3 P
8 3 I R 8 2 Q
9 3 C A 9 3 P
4 8 C T 7 4 P
9 3 T G 9 3 P

Desired output:

A B C D E F G
1 2 T TACA 3 2 Q
9 3 A C 9 3 P
8 3 I R 8 2 Q
4 8 C T 7 4 P
9 3 T G 9 3 P

Using: sort -k3,4 -k5,5r -k1,1r file | sort -k1,1 -k3,4 -u | sort -k1,1r (or similar) keeps both the 9 3 A C 9 3 P and 9 3 C A 9 3 P versions but I only want to keep one of these. The added complication is that for values for Column A could have multiple acceptable Column C and D values (such as row 2 9 3 A C 9 3 P and row 5 9 3 T G 9 3 P in the Desired Output example above) so cannot search for duplicates based on Column A alone.

Thanks!

0
awk '{
    key1 = $1 FS $2 FS $5 FS $5 FS $7
    if ( ((key1 SUBSEP $3 FS $4) in seen) || ((key1 SUBSEP $4 FS $3) in seen) )
        next
    seen[key1, $3 FS $4] = 1
    print
}'
A B C D E F G
1 2 T TACA 3 2 Q
9 3 A C 9 3 P
8 3 I R 8 2 Q
4 8 C T 7 4 P
9 3 T G 9 3 P

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.