I have a file A of lines of equal length n. I have a program that for each line in A can output a line of length n which its key. The program can either write the sequence of keys for the lines in A into a file B or write them to stdout for each line of A.

I need to obtain all lines of A so that the respective keys do not get repeated.. What would be the most efficient way to accomplish this? I see that there is a -k option in sort but it doesn't quite seem to do what I want.

Edit. To clarify on the comment.

Suppose that A contains the lines


and that the second program computing the keys for these lines outputs


where the first line is the otput for foo, the second for bar and third for baz.

Then one of the correct outputs would be

  • Please clarify what you meant by "which its key", and provide sample data (input and expected output for the given input).
    – Janis
    Mar 16, 2015 at 15:26
  • 1
    foo shouldn't be in the output as it doesn't have an unique key... or you should change your question.
    – wurtel
    Mar 16, 2015 at 15:33
  • @wurtel Hopefully the question is now correctly phrased.
    – Jernej
    Mar 17, 2015 at 9:11

3 Answers 3


As you have specified (but consider wurtel's comment on your question):

generate_keys A > B   # adjust this call however that program is defined to work

awk '
  NR==FNR { a[NR]=$1 ; next }
  !b[$1]++ { print a[FNR] }
' A B
  • If using bash, don't need the temp file: awk '...' A <(generate_keys A) Mar 16, 2015 at 15:55
  • 1
    @glennjackman, or generate_keys A | awk '...' A -, no need for process substitution (a ksh feature, also available in zsh and bash btw). Mar 16, 2015 at 15:57

can't test without some sample data, but something like this:

paste <(generate-keys "$filename") "$filename" |
awk '! seen[$1]++ {print $2}'
  • 1
    This is assuming there are no spaces in the lines.
    – wurtel
    Mar 16, 2015 at 15:32
  • True. A common delimiter can be declared for paste and awk, if a delimiter not appearing in the data is apparent. Mar 16, 2015 at 15:34
  • Are you sure there is not a missing " somewhere in the above script?
    – Jernej
    Mar 16, 2015 at 15:44
$ getkeys A > B
$ sort B | uniq -c | awk '{if($1 == 1) print $2}' > C
$ paste B A | fgrep -f C | cut -f2-


First generate the keys for each line.

Then count the number of times each key occurs, and save those keys that occur once in file C.

Then join the keys with the lines using paste, match those lines with the list of unique keys with fgrep, then select just the line (omitting the key) using cut.

It might be better to let the strings in C begin with a ^ and to use grep instead of fgrep so that it only matches the beginning of the line, i.e. the key, but if it's anything like an md5sum then chances of false matches will be small. (And I'm lazy :-) )

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.