lines with unique checksum

Question

I have a file A of lines of equal length n. I have a program that for each line in A can output a line of length n which its key. The program can either write the sequence of keys for the lines in A into a file B or write them to stdout for each line of A.

I need to obtain all lines of A so that the respective keys do not get repeated.. What would be the most efficient way to accomplish this? I see that there is a -k option in sort but it doesn't quite seem to do what I want.

Edit. To clarify on the comment.

Suppose that A contains the lines

foo
bar 
baz

and that the second program computing the keys for these lines outputs

xxx
yyy
xxx

where the first line is the otput for foo, the second for bar and third for baz.

Then one of the correct outputs would be

foo
bar

Answer 1

As you have specified (but consider wurtel's comment on your question):

generate_keys A > B   # adjust this call however that program is defined to work

awk '
  NR==FNR { a[NR]=$1 ; next }
  !b[$1]++ { print a[FNR] }
' A B

Answer 2

can't test without some sample data, but something like this:

paste <(generate-keys "$filename") "$filename" |
awk '! seen[$1]++ {print $2}'

Answer 3

$ getkeys A > B
$ sort B | uniq -c | awk '{if($1 == 1) print $2}' > C
$ paste B A | fgrep -f C | cut -f2-

Explanation:

First generate the keys for each line.

Then count the number of times each key occurs, and save those keys that occur once in file C.

Then join the keys with the lines using paste, match those lines with the list of unique keys with fgrep, then select just the line (omitting the key) using cut.

It might be better to let the strings in C begin with a ^ and to use grep instead of fgrep so that it only matches the beginning of the line, i.e. the key, but if it's anything like an md5sum then chances of false matches will be small. (And I'm lazy :-) )