1

I have a single ascii file (Report.dat) with these lines:

a  ./L1/file2.txt
c  ./L1/file1.txt
b  ./L2/file1.txt
a  ./L2/file2.txt

and I need to get just the lines where the first column is unique, as follows:

c  ./L1/file1.txt
b  ./L2/file1.txt

How can I do that using AWK, sort or uniq commands ?

  • opposite of unix.stackexchange.com/questions/394731/… – Sundeep Mar 8 '18 at 11:50
  • does the 1st column always consist of 1 char? – RomanPerekhrest Mar 8 '18 at 11:59
  • @RomanPerekhrest - no, it's in fact a long alphanumeric string. – LeMike Mar 8 '18 at 12:07
  • Excellent reference @Sundeep, yes ,it seems to be the opposite. :) – LeMike Mar 8 '18 at 12:16
  • 1
    @LeMike but is it a fixed length alphanumeric string? if it is, then at least with GNU uniq you could use sort Report.dat | uniq -uwN where N is the length of the string – steeldriver Mar 8 '18 at 13:15
1

This is how you get the "labels" that are unique:

$ awk '{ print $1 }' Report.dat | sort | uniq -u
b
c

These can be converted to regular expressions that match at the start of the line by adding ^ in front of the strings:

$ awk '{ print $1 }' Report.dat | sort | uniq -u | sed 's/^/^/'
^b
^c

You can apply these regular expressions to the original file to get the corresponding lines:

$ awk '{ print $1 }' Report.dat | sort | uniq -u | sed 's/^/^/' | grep -f /dev/stdin Report.dat
c  ./L1/file1.txt
b  ./L2/file1.txt

With grep -f /dev/stdin we tell grep to take the regular expressions coming from sed and use these to do matching in Report.dat.


Alternatively, to bypass the sed step:

awk '{ print "^" $1 }' Report.dat | sort | uniq -u | grep -f /dev/stdin Report.dat

Alternatively, do it all in awk:

awk 'NR == FNR { c[$1]++; next } c[$1] == 1' Report.dat Report.dat

This reads the file twice. The first time around, it simply counts the number of time that each "label" occurs. The second time around, it tests the label on the current line to see if its count is one before printing the line.

Note that there is no way around parsing the file twice. One could store the complete file in memory and parse it twice there, but that's fiddly and will be problematic in the general case when we don't know how large files we are feeding into the script.

  • Excellent!!! @Kusalananda! It works fine!, and could be possible to show the lines where the first column exists more than once too ? – LeMike Mar 8 '18 at 11:49
  • @LeMike This would possibly be simplest with the last variation using only awk. Just change == to >. – Kusalananda Mar 8 '18 at 11:50
  • 1
    Thanks very much @Kusalananda!, I'm very much grateful for your help!. – LeMike Mar 8 '18 at 12:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.