0

This question already has an answer here:

Example of file with duplicate lines:

 # more  file.csv

 spark2-env,spark_thrift_cmd_opts,--driver-memory 15g --executor-cores 7
 spark2-thrift-sparkconf,spark.memory.offHeap.enabled,true
 mapred-site,yarn.app.mapreduce.am.command-opts,-Xmx5324m -Dhdp.version=${hdp.version}
 spark2-env,spark_thrift_cmd_opts,--driver-memory 15g --executor-cores 7

We created the following awk in order to delete the lines with duplicate strings only in fields 1 and 2, but awk from some unclear reason also deletes the line where field 3 is duplicate.

What is wrong with my awk syntax?

 # awk -F, 'NR==FNR{a[$1,$2]++; next} a[$1,$2]==1'  blueprint_param_file.csv  blueprint_param_file.csv

 spark2-thrift-sparkconf,spark.memory.offHeap.enabled,true
 mapred-site,yarn.app.mapreduce.am.command-opts,-Xmx5324m -Dhdp.version=\${hdp.version}

Expected output:

spark2-env,spark_thrift_cmd_opts,--driver-memory 15g --executor-cores 7
spark2-thrift-sparkconf,spark.memory.offHeap.enabled,true
mapred-site,yarn.app.mapreduce.am.command-opts,-Xmx5324m -    

marked as duplicate by Philippos, Stephen Rauch, Satō Katsura, Romeo Ninov, Kusalananda Aug 29 '17 at 16:18

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    You already asked this. In how does your new question differ from the one that was answered? – Philippos Aug 29 '17 at 14:26
  • no duplicate just we found bug in the previous answer – yael Aug 29 '17 at 14:34
  • How about sort -t, -suk1,2 – Stéphane Chazelas Aug 29 '17 at 15:31
  • This is a Q&A site. If there is a bug in previous answer, it needs to get fixed there! – Philippos Aug 29 '17 at 17:57
3

No need to process the same file twice, it's enough to apply the following logic:

awk -F, '!a[$1,$2]++' blueprint_param_file.csv
  • !a[$1,$2]++ - ensures that a crucial key(concatenation of the 1st and 2nd field) occurs only once in array a, thereby excluding duplicates

The output:

spark2-env,spark_thrift_cmd_opts,--driver-memory 15g --executor-cores 7
spark2-thrift-sparkconf,spark.memory.offHeap.enabled,true
mapred-site,yarn.app.mapreduce.am.command-opts,-Xmx5324m -Dhdp.version=${hdp.version}

Not the answer you're looking for? Browse other questions tagged or ask your own question.