1

I have a file A of lines of equal length n. I have a program that for each line in A can output a line of length n which its key. The program can either write the sequence of keys for the lines in A into a file B or write them to stdout for each line of A.

I need to obtain all lines of A so that the respective keys do not get repeated.. What would be the most efficient way to accomplish this? I see that there is a -k option in sort but it doesn't quite seem to do what I want.

Edit. To clarify on the comment.

Suppose that A contains the lines

foo
bar 
baz

and that the second program computing the keys for these lines outputs

xxx
yyy
xxx

where the first line is the otput for foo, the second for bar and third for baz.

Then one of the correct outputs would be

foo
bar
Improve this question
3
  • Please clarify what you meant by "which its key", and provide sample data (input and expected output for the given input). – Janis Mar 16 '15 at 15:26
  • 1
    foo shouldn't be in the output as it doesn't have an unique key... or you should change your question. – wurtel Mar 16 '15 at 15:33
  • @wurtel Hopefully the question is now correctly phrased. – Jernej Mar 17 '15 at 9:11
2

can't test without some sample data, but something like this:

paste <(generate-keys "$filename") "$filename" |
awk '! seen[$1]++ {print $2}'
Improve this answer
3
  • 1
    This is assuming there are no spaces in the lines. – wurtel Mar 16 '15 at 15:32
  • True. A common delimiter can be declared for paste and awk, if a delimiter not appearing in the data is apparent. – glenn jackman Mar 16 '15 at 15:34
  • Are you sure there is not a missing " somewhere in the above script? – Jernej Mar 16 '15 at 15:44
3

As you have specified (but consider wurtel's comment on your question):

generate_keys A > B   # adjust this call however that program is defined to work

awk '
  NR==FNR { a[NR]=$1 ; next }
  !b[$1]++ { print a[FNR] }
' A B
Improve this answer
2
  • If using bash, don't need the temp file: awk '...' A <(generate_keys A) – glenn jackman Mar 16 '15 at 15:55
  • 1
    @glennjackman, or generate_keys A | awk '...' A -, no need for process substitution (a ksh feature, also available in zsh and bash btw). – Stéphane Chazelas Mar 16 '15 at 15:57
2 
$ getkeys A > B
$ sort B | uniq -c | awk '{if($1 == 1) print $2}' > C
$ paste B A | fgrep -f C | cut -f2-

Explanation:

First generate the keys for each line.

Then count the number of times each key occurs, and save those keys that occur once in file C.

Then join the keys with the lines using paste, match those lines with the list of unique keys with fgrep, then select just the line (omitting the key) using cut.

It might be better to let the strings in C begin with a ^ and to use grep instead of fgrep so that it only matches the beginning of the line, i.e. the key, but if it's anything like an md5sum then chances of false matches will be small. (And I'm lazy :-) )

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.