I need to remove both duplicates from file test.txt
based on first 2 columns:
tom,22,777
tom,22,888
tom,18,54
ray,44,43843
expected result:
tom,18,54
ray,44,43843
I tried using cat test.txt | sort -u t, k1,1 k2,2
but it is not working
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this communityI need to remove both duplicates from file test.txt
based on first 2 columns:
tom,22,777
tom,22,888
tom,18,54
ray,44,43843
expected result:
tom,18,54
ray,44,43843
I tried using cat test.txt | sort -u t, k1,1 k2,2
but it is not working
You could use awk
: process the file twice, once to count the occurrences of 1st and 2nd fields and then again to print lines if count is 1:
awk -F, 'NR==FNR{seen[$1,$2]++;next};seen[$1,$2]==1' infile infile
One way to do it, assuming GNU grep(1)
(for fgrep -f -
), and assuming fields in your input consist only of alphanumeric characters:
cut -d , -f 1,2 file.csv | sort | uniq -d | fgrep -vwf - file.csv
grep(1)
. Or you you have some alias messing with results.
Dec 13, 2016 at 11:44
Try This
cat file.txt | grep -v "`cut -d , -f 1,2 file.txt | uniq -d`"