1

I have the input file (sorted by column 2 with -t,):

TOP,25424242,T0137,0.08,0.06,0.02,24
TOP,25424242,T0138,0.07,0.06,0.01,24
TOP,17236110,T0138,9.65,9.37,0.28,89
TOP,23525255,T0137,0.40,0.30,0.11,24
TOP,23525255,T0138,0.08,0.07,0.01,24
TOP,21627012,T0138,0.41,0.33,0.08,24
TOP,75856354,T0137,0.18,0.17,0.01,36
TOP,75856354,T0138,0.18,0.17,0.01,26
TOP,42401990,T0137,0.06,0.05,0.01,24

I want to get rid of every two lines which have the same value in column 2, so finally get only lines with unique value in field 2 - from example above it would be:

TOP,17236110,T0138,9.65,9.37,0.28,89
TOP,21627012,T0138,0.41,0.33,0.08,24
TOP,42401990,T0137,0.06,0.05,0.01,24
1
  • 1
    and what did you try so far? Dec 21 '17 at 9:56
2

This should work:

 $ awk -F, '{a[$2]=$0; b[$2]++;} END{for(i in a){if(b[i]==1){print a[i]}}}' file
TOP,17236110,T0138,9.65,9.37,0.28,89
TOP,21627012,T0138,0.41,0.33,0.08,24
TOP,42401990,T0137,0.06,0.05,0.01,24
2
  • thank you, it works on example but when I process huge file (same sorted by second column, finally when processed your awk code output is unsored in column 2. I can't explain why it is not processed one-by-one, numbers in second column should be incremental in column 2 as it was before processing...
    – DonJ
    Dec 21 '17 at 10:23
  • @DonJ this is why you need to always mention all of your requirements in the original question. You didn't say anything about sorting and awk iterates over arrays in random order. Just pass the output through | sort -t, -k2.
    – terdon
    Dec 21 '17 at 10:25
2

Short uniq trick for your current input structure(with static first 2 fields length):

uniq -s4 -w8 -u file
  • -s4 - skip the first 4 chars (i.e. TOP,)
  • -w8 - compare no more than 8 characters in lines
  • -u - only print unique lines

The output:

TOP,17236110,T0138,9.65,9.37,0.28,89
TOP,21627012,T0138,0.41,0.33,0.08,24
TOP,42401990,T0137,0.06,0.05,0.01,24
0
-1

You can achieve this using awk:

for k in `awk -F "," '{print $2}' file.txt | uniq -D`; do
  sed -i '/'$k'/d' file.txt;
done

Output

TOP,17236110,T0138,9.65,9.37,0.28,89
TOP,21627012,T0138,0.41,0.33,0.08,24
TOP,42401990,T0137,0.06,0.05,0.01,24
1
  • Please copy/paste that into shellcheck.net so it can tell you about some of the issues but also that will fail when key values can be substrings of other key values and/or appear elsewhere in the input so it's a bad approach plus if you were going to do this it could be done with grep -f or other better ways.
    – Ed Morton
    Sep 21 at 17:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.