I need to remove both duplicates from file test.txt based on first 2 columns:
tom,22,777
tom,22,888
tom,18,54
ray,44,43843
expected result:
tom,18,54
ray,44,43843
I tried using cat test.txt | sort -u t, k1,1 k2,2 but it is not working
Add a comment
|
3 Answers
You could use awk: process the file twice, once to count the occurrences of 1st and 2nd fields and then again to print lines if count is 1:
awk -F, 'NR==FNR{seen[$1,$2]++;next};seen[$1,$2]==1' infile infile
One way to do it, assuming GNU grep(1) (for fgrep -f -), and assuming fields in your input consist only of alphanumeric characters:
cut -d , -f 1,2 file.csv | sort | uniq -d | fgrep -vwf - file.csv
answered Dec 13, 2016 at 11:05
-
cut -d , -f 1,2 file.csv | sort | uniq -u | fgrep -vwf - file.csv is removing the duplicats but i am getting the result without the third field @ Sato Katsura– WAELDec 13, 2016 at 11:36
-
2@WAEL Then you aren't using GNU
grep(1). Or you you have some alias messing with results. Dec 13, 2016 at 11:44 -
@ Sato when i past the same query you provided cut -d , -f 1,2 test.txt | sort | uniq -d | fgrep -vwf - test.txt i will get error "fgrep: 0652-033 Cannot open-."– WAELDec 13, 2016 at 11:56
-
-
@ Sato how can I check whether i am using GNU grep(1) thanks for your support– WAELDec 13, 2016 at 12:52
Try This
cat file.txt | grep -v "`cut -d , -f 1,2 file.txt | uniq -d`"
-
Dear Prince i am getting "grep: 0652-226 Maximum line length of 8192 exceeded." once the file is getting bigger how can i resolve this issue– WAELDec 24, 2016 at 12:00