4

For a 3 column csv file, list.csv, how would you remove subsequent duplicate rows where the 1st field matches, and just the first 3 chars of the 2nd field match? Some rows will have a 2nd field with less than 3 chars.

list.csv:

12,12345,a
12,12345,b
123,12345,a
1234,12,b
1234,12345,a
567,567,a
567,56712,a
567,56734,a
567,6789,a

Expected output:

12,12345,a
123,12345,a
1234,12,b
1234,12345,a
567,567,a
567,6789,a
8

sort should work as well

 sort -t, -k1,1 -k2.1,2.3 -u <list.csv
 12,12345,a
 123,12345,a
 1234,12,b
 1234,12345,a
 567,567,a
 567,6789,a
5
awk -F, '!seen[$1,substr($2,1,3)]++' list.csv 
12,12345,a
123,12345,a
1234,12,b
1234,12345,a
567,567,a
567,6789,a

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.