How to remove both duplicates from file based on the first 2 columns

Question

I need to remove both duplicates from file test.txt based on first 2 columns:

tom,22,777
tom,22,888
tom,18,54
ray,44,43843

expected result:

tom,18,54
ray,44,43843

I tried using cat test.txt | sort -u t, k1,1 k2,2 but it is not working

don_crissti · Accepted Answer · 2016-12-13 13:25:34Z

1

You could use awk: process the file twice, once to count the occurrences of 1st and 2nd fields and then again to print lines if count is 1:

awk -F, 'NR==FNR{seen[$1,$2]++;next};seen[$1,$2]==1' infile infile

answered Dec 13 '16 at 13:25

community wiki

Add a comment |

Satō Katsura · Accepted Answer · 2016-12-13 11:05:02Z

1

One way to do it, assuming GNU grep(1) (for fgrep -f -), and assuming fields in your input consist only of alphanumeric characters:

cut -d , -f 1,2 file.csv | sort | uniq -d | fgrep -vwf - file.csv

answered Dec 13 '16 at 11:05

Satō Katsura

12.5k2 gold badges25 silver badges46 bronze badges

cut -d , -f 1,2 file.csv | sort | uniq -u | fgrep -vwf - file.csv is removing the duplicats but i am getting the result without the third field @ Sato Katsura
– WAEL
Dec 13 '16 at 11:36
2

@WAEL Then you aren't using GNU grep(1). Or you you have some alias messing with results.
– Satō Katsura
Dec 13 '16 at 11:44
@ Sato when i past the same query you provided cut -d , -f 1,2 test.txt | sort | uniq -d | fgrep -vwf - test.txt i will get error "fgrep: 0652-033 Cannot open-."
– WAEL
Dec 13 '16 at 11:56
@WAEL Then you aren't using GNU grep(1).
– Satō Katsura
Dec 13 '16 at 11:59
@ Sato how can I check whether i am using GNU grep(1) thanks for your support
– WAEL
Dec 13 '16 at 12:52

| Show 1 more comment

prince 987 · Accepted Answer · 2016-12-13 13:21:47Z

1

Try This

 cat file.txt | grep -v  "`cut -d , -f 1,2 file.txt | uniq -d`"

answered Dec 13 '16 at 13:21

prince 987

741 silver badge7 bronze badges

Dear Prince i am getting "grep: 0652-226 Maximum line length of 8192 exceeded." once the file is getting bigger how can i resolve this issue
– WAEL
Dec 24 '16 at 12:00

Add a comment |

3 Answers 3