0

I have a CSV file. It is ordered by an identifier the first field. The identifier in the first field can appear in either 1 or in 2 lines in the field. I want to remove all lines in the file where the identifier only occurs once.

Example, file contains:

Id1,x11,x12...
Id2,x21,x22...
Id2,x23,x24...

In this case, I want to remove the first line because Id1 only occurs once.

  • Is the file sorted by the identified field or not? – thrig Apr 11 '16 at 17:02
1

Give this a try:

 awk 'BEGIN{FS=",";}{if (!ids[$1]) {ids[$1]=1;lines[$1]=$0;} else {if (ids[$1]==1) {print lines[$1]; print; ids[$1]++;} else {print;}}}'

When an id is encountered first, it is stored into the ids associative array, and the whole line is stored into lines associative array but not printed.

When an id is encountered twice or more, the line stored is printed and each additional lines are printed as well.

The test:

$ cat data.csv
Id1,x11,x12...
Id2,x21,x22...
Id2,x23,x24...

$  awk 'BEGIN{FS=",";}{if (!ids[$1]) {ids[$1]=1;lines[$1]=$0;} else {if (ids[$1]==1) {print lines[$1]; print; ids[$1]++;} else {print;}}}' data.csv
Id2,x21,x22...
Id2,x23,x24...
1

Jay's answer is the general awk solution for unordered, but for ordered it can be simpler:

awk -F, '$1!=id {id=$1;pr=$0 ORS;next} {print pr $0;pr=""}'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.