9

I have the following file:

AA,true
AA,false
BB,false
CC,false
BB,true
DD,true

I am trying to look for duplicates and remove the line that has the column value equals to true.

as output it should be:

AA,false
BB,false
CC,false
DD,true
  • 2
    So.. only keep true if it's the first instance of the first column? – DopeGhoti Jul 20 '17 at 21:18
  • 1
    @RomanPerekhrest Probably because is a uniq entry and is printed "as it is" – George Vasiliou Jul 20 '17 at 21:30
  • @RomanPerekhrest because DD,true is not a duplicate we don't have another line with DD,false. – Hani Gotc Jul 20 '17 at 21:34
  • AA,true AA,false AA,false AA,false What output should be in this case? I understand, that row should be removed only if it is having duplicate and contain true at the same time. All false rows should be stay untouched in any case. That is, in this case, only AA, true will be removed. But all answers leaves only one line - AA,false. Just interesting :) – MiniMax Jul 21 '17 at 21:08
9
awk -F, '$2 == "false" {data[$1]=$2 } $2=="true" { if ( data[$1]!="false" ) { data[$1]=$2 } } END { OFS=","; for (item in data) { print item,data[item] }}' input

To expand the script vertically for explanation:

BEGIN {
   FS=","         # Set the input separator; this is what -F, does.
}
$2 == "false" {    # For any line whose second field is "false", we
   data[$1]=$2     # will use that value no matter what.
}
$2=="true" {                    # For lines whose second field is "true",
   if ( data[$1]!="false" ) {   # only keep if if we haven't yet seen a
      data[$1]=$2               # "false"
   }
}
END {                           # Now that we have tabulated our data, we
   OFS=","                      # can print it out by iterating through 
   for (item in data) {         # the array we created.
      print item,data[item]
   }
}
  • @DopeGhoti well explained! You've got my +1 on this. – Valentin Bajrami Jul 20 '17 at 21:30
14

Simple version:

sort input.txt | awk -F, '!a[$1]++'

"false" sorts alphabetically before "true," and the Awk command here just keeps the first row only for each distinct first field value.

If you want to keep "true" instead of "false," reverse sort it, pass it to the same Awk command, and reverse sort it again afterward.

  • 1
    also, if -u option is available, sort input.txt | sort -t, -u -k1,1 – Sundeep Jul 21 '17 at 7:58
  • 2
    @Sundeep why use two sort calls? Why not just sort -ut, -k1,1 input.txt ? – terdon Jul 21 '17 at 14:59
  • 2
    @terdon because -u will retain the first line found from input file among duplicates... for given case, input has to be sorted before -u can be applied... for ex: AA,true will be printed instead of AA,false since it appears first in given sample.. same reason why awk -F, '!a[$1]++' alone won't solve this problem – Sundeep Jul 21 '17 at 15:18
5
perl -F, -lane '
   exists $h{$F[0]} or $h[$h{$F[0]}=@h]=$_;
   $h=$_; /,false$/ or $_=$h for $h[$h{$F[0]}];
   END{ print for @h; }
' duplicates.file

Data structures:

  • Hash %h whose keys are first fields (AAA, BBB, CCC, etc.) and corresponding values are numbers telling the order in which the keys were encountered. Thus, e.g., key AAA => 0, key BBB => 1, key CCC => 2.
  • Array @h whose elements are lines contained in the order of printing. So if both true and false are found in data, then the false value will go into the array. OTW, if there's one type of data, then that would be present.

Another way is using GNU sed:

sed -Ee '
   G
   /^([^,]*),(false|true)\n(.*\n)?\1,\2(\n|$)/ba
   /^([^,]*)(,true)\n(.*\n)?\1,false(\n|$)/ba
   /^([^,]*)(,false)\n((.*\n)?)\1,true(\n|$)/{
      s//\3\1\2\5/;h;ba
   }
   s/([^\n]*)\n(.*)$/\2\n\1/;s/^\n*//
   h;:a;$!d;g
' duplicates.file

FWIW, the POSIX equivalent code for the above GNU-sed code is listed below:

sed -e '
   G

   /^\([^,]*\),\(false\)\n\(.*\n\)\{0,1\}\1,\2$/ba
   /^\([^,]*\),\(false\)\n\(.*\n\)\{0,1\}\1,\2\n/ba

   /^\([^,]*\),\(true\)\n\(.*\n\)\{0,1\}\1,\2$/ba
   /^\([^,]*\),\(true\)\n\(.*\n\)\{0,1\}\1,\2\n/ba

   /^\([^,]*\),true\n\(.*\n\)\{0,1\}\1,false$/ba
   /^\([^,]*\),true\n\(.*\n\)\{0,1\}\1,false\n/ba

   /^\([^,]*\)\(,false\)\n\(\(.*\n\)\{0,1\}\)\1,true$/{
      s//\3\1\2/
      h
      ba
   }
   /^\([^,]*\)\(,false\)\n\(\(.*\n\)\{0,1\}\)\1,true\n/{
      s//\3\1\2\n/
      h
      ba
   }

   y/\n_/_\n/
   s/\([^_]*\)_\(.*\)$/\2_\1/;s/^_*//
   y/\n_/_\n/

   h;:a;$!d;g
' duplicates.file

Explanation

  • In this method we store the result to be finally printed in the hold space.
  • For every line read, we append the hold space to the pattern space for examination of the current line vis-a-vis the existing state of the hold space.
  • Now 5 things can possibly happen during this comparison:
    • a) Current line matches somewhere in the hold line & false:false.
      • [ACTION] Since same false state is found, then do nothing.
    • b) Current line matches somewhere in the hold line & true:true.
      • [ACTION] Since same true state is found, then do nothing.
    • c) Current line matches somewhere in the hold line & true:false.
      • [ACTION] Since a false state already exists, do nothing.
    • d) Current line matches somewhere in the hold line & false:true.
      • [ACTION] This involves some work, in that we need to replace the false line at the exact same position where the true is located.
    • e) Current line DOES NOT match anywhere in the hold line.
      • [ACTION] Move the present line to the end.

Results

AA,false
BB,false
CC,false
DD,true
3

For each input line, store the value of the second field in associative array a (using the first field as the array's key) ONLY if we haven't already stored the value false for that key. Use , for both input and output field separator. Print out the array after we've read all input lines.

$ awk -F, -v OFS=, 'a[$1] != "false" { a[$1] = $2 };
                    END { for (i in a) {print i,a[i]} }' truefalse.txt
AA,false
BB,false
CC,false
DD,true

The significant difference between this and DopeGhoti's version is that this version doesn't care at all about the value of $2, it only cares about the value, if any, of a[$1].

1

Two-pass sort solution

sort -k1,1 -k2,2 -t, file | sort -k1,1 -t, -u

First sort pass clusters records by field 1 with false records preceding true for each block of records sharing a common field 1 value. The second sort pass is set up to yield one record for each distinct value within field 1 courtesy the -u. Since -u implies stable sort, the one record thus yielded is the first record encountered for each distinct value within field 1 - which is a record with false in the second field due to the work done by the first sort pass

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.