How to get lines from unique columns?

Question

I have a single ascii file (Report.dat) with these lines:

a  ./L1/file2.txt
c  ./L1/file1.txt
b  ./L2/file1.txt
a  ./L2/file2.txt

and I need to get just the lines where the first column is unique, as follows:

c  ./L1/file1.txt
b  ./L2/file1.txt

How can I do that using AWK, sort or uniq commands ?

Answer 1

This is how you get the "labels" that are unique:

$ awk '{ print $1 }' Report.dat | sort | uniq -u
b
c

These can be converted to regular expressions that match at the start of the line by adding ^ in front of the strings:

$ awk '{ print $1 }' Report.dat | sort | uniq -u | sed 's/^/^/'
^b
^c

You can apply these regular expressions to the original file to get the corresponding lines:

$ awk '{ print $1 }' Report.dat | sort | uniq -u | sed 's/^/^/' | grep -f /dev/stdin Report.dat
c  ./L1/file1.txt
b  ./L2/file1.txt

With grep -f /dev/stdin we tell grep to take the regular expressions coming from sed and use these to do matching in Report.dat.

---

Alternatively, to bypass the sed step:

awk '{ print "^" $1 }' Report.dat | sort | uniq -u | grep -f /dev/stdin Report.dat

---

Alternatively, do it all in awk:

awk 'NR == FNR { c[$1]++; next } c[$1] == 1' Report.dat Report.dat

This reads the file twice. The first time around, it simply counts the number of time that each "label" occurs. The second time around, it tests the label on the current line to see if its count is one before printing the line.

Note that there is no way around parsing the file twice. One could store the complete file in memory and parse it twice there, but that's fiddly and will be problematic in the general case when we don't know how large files we are feeding into the script.