1

I need to remove both duplicates from file test.txt based on first 2 columns:

tom,22,777
tom,22,888
tom,18,54
ray,44,43843

expected result:

tom,18,54
ray,44,43843

I tried using cat test.txt | sort -u t, k1,1 k2,2 but it is not working

0
1

One way to do it, assuming GNU grep(1) (for fgrep -f -), and assuming fields in your input consist only of alphanumeric characters:

cut -d , -f 1,2 file.csv | sort | uniq -d | fgrep -vwf - file.csv
6
  • cut -d , -f 1,2 file.csv | sort | uniq -u | fgrep -vwf - file.csv is removing the duplicats but i am getting the result without the third field @ Sato Katsura
    – WAEL
    Dec 13 '16 at 11:36
  • 2
    @WAEL Then you aren't using GNU grep(1). Or you you have some alias messing with results. Dec 13 '16 at 11:44
  • @ Sato when i past the same query you provided cut -d , -f 1,2 test.txt | sort | uniq -d | fgrep -vwf - test.txt i will get error "fgrep: 0652-033 Cannot open-."
    – WAEL
    Dec 13 '16 at 11:56
  • @WAEL Then you aren't using GNU grep(1). Dec 13 '16 at 11:59
  • @ Sato how can I check whether i am using GNU grep(1) thanks for your support
    – WAEL
    Dec 13 '16 at 12:52
1

Try This

 cat file.txt | grep -v  "`cut -d , -f 1,2 file.txt | uniq -d`"
1
  • Dear Prince i am getting "grep: 0652-226 Maximum line length of 8192 exceeded." once the file is getting bigger how can i resolve this issue
    – WAEL
    Dec 24 '16 at 12:00
1

You could use awk: process the file twice, once to count the occurrences of 1st and 2nd fields and then again to print lines if count is 1:

awk -F, 'NR==FNR{seen[$1,$2]++;next};seen[$1,$2]==1' infile infile
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.