I have the following file:
AA,true
AA,false
BB,false
CC,false
BB,true
DD,true
I am trying to look for duplicates and remove the line that has the column value equals to true
.
as output it should be:
AA,false
BB,false
CC,false
DD,true
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this communityI have the following file:
AA,true
AA,false
BB,false
CC,false
BB,true
DD,true
I am trying to look for duplicates and remove the line that has the column value equals to true
.
as output it should be:
AA,false
BB,false
CC,false
DD,true
awk -F, '$2 == "false" {data[$1]=$2 } $2=="true" { if ( data[$1]!="false" ) { data[$1]=$2 } } END { OFS=","; for (item in data) { print item,data[item] }}' input
To expand the script vertically for explanation:
BEGIN {
FS="," # Set the input separator; this is what -F, does.
}
$2 == "false" { # For any line whose second field is "false", we
data[$1]=$2 # will use that value no matter what.
}
$2=="true" { # For lines whose second field is "true",
if ( data[$1]!="false" ) { # only keep if if we haven't yet seen a
data[$1]=$2 # "false"
}
}
END { # Now that we have tabulated our data, we
OFS="," # can print it out by iterating through
for (item in data) { # the array we created.
print item,data[item]
}
}
Simple version:
sort input.txt | awk -F, '!a[$1]++'
"false" sorts alphabetically before "true," and the Awk command here just keeps the first row only for each distinct first field value.
If you want to keep "true" instead of "false," reverse sort it, pass it to the same Awk command, and reverse sort it again afterward.
sort
calls? Why not just sort -ut, -k1,1 input.txt
?
– terdon♦
Jul 21 '17 at 14:59
-u
will retain the first line found from input file among duplicates... for given case, input has to be sorted before -u
can be applied... for ex: AA,true
will be printed instead of AA,false
since it appears first in given sample.. same reason why awk -F, '!a[$1]++'
alone won't solve this problem
– Sundeep
Jul 21 '17 at 15:18
perl -F, -lane '
exists $h{$F[0]} or $h[$h{$F[0]}=@h]=$_;
$h=$_; /,false$/ or $_=$h for $h[$h{$F[0]}];
END{ print for @h; }
' duplicates.file
Data structures:
%h
whose keys are first fields (AAA, BBB, CCC, etc.) and corresponding values are numbers telling the order in which the keys were encountered. Thus, e.g., key AAA => 0, key BBB => 1, key CCC => 2.@h
whose elements are lines contained in the order of printing. So if both true and false are found in data, then the false value will go into the array. OTW, if there's one type of data, then that would be present.Another way is using GNU sed:
sed -Ee '
G
/^([^,]*),(false|true)\n(.*\n)?\1,\2(\n|$)/ba
/^([^,]*)(,true)\n(.*\n)?\1,false(\n|$)/ba
/^([^,]*)(,false)\n((.*\n)?)\1,true(\n|$)/{
s//\3\1\2\5/;h;ba
}
s/([^\n]*)\n(.*)$/\2\n\1/;s/^\n*//
h;:a;$!d;g
' duplicates.file
FWIW, the POSIX equivalent code for the above GNU-sed code is listed below:
sed -e '
G
/^\([^,]*\),\(false\)\n\(.*\n\)\{0,1\}\1,\2$/ba
/^\([^,]*\),\(false\)\n\(.*\n\)\{0,1\}\1,\2\n/ba
/^\([^,]*\),\(true\)\n\(.*\n\)\{0,1\}\1,\2$/ba
/^\([^,]*\),\(true\)\n\(.*\n\)\{0,1\}\1,\2\n/ba
/^\([^,]*\),true\n\(.*\n\)\{0,1\}\1,false$/ba
/^\([^,]*\),true\n\(.*\n\)\{0,1\}\1,false\n/ba
/^\([^,]*\)\(,false\)\n\(\(.*\n\)\{0,1\}\)\1,true$/{
s//\3\1\2/
h
ba
}
/^\([^,]*\)\(,false\)\n\(\(.*\n\)\{0,1\}\)\1,true\n/{
s//\3\1\2\n/
h
ba
}
y/\n_/_\n/
s/\([^_]*\)_\(.*\)$/\2_\1/;s/^_*//
y/\n_/_\n/
h;:a;$!d;g
' duplicates.file
AA,false
BB,false
CC,false
DD,true
For each input line, store the value of the second field in associative array a
(using the first field as the array's key) ONLY if we haven't already stored the value false
for that key. Use ,
for both input and output field separator. Print out the array after we've read all input lines.
$ awk -F, -v OFS=, 'a[$1] != "false" { a[$1] = $2 };
END { for (i in a) {print i,a[i]} }' truefalse.txt
AA,false
BB,false
CC,false
DD,true
The significant difference between this and DopeGhoti's version is that this version doesn't care at all about the value of $2
, it only cares about the value, if any, of a[$1]
.
Two-pass sort
solution
sort -k1,1 -k2,2 -t, file | sort -k1,1 -t, -u
First sort
pass clusters records by field 1
with false
records preceding true
for each block of records sharing a common field 1
value. The second sort
pass is set up to yield one record for each distinct value within field 1
courtesy the -u
. Since -u
implies stable sort, the one record thus yielded is the first record encountered for each distinct value within field 1
- which is a record with false
in the second field due to the work done by the first sort
pass
true
if it's the first instance of the first column? – DopeGhoti Jul 20 '17 at 21:18AA,true AA,false AA,false AA,false
What output should be in this case? I understand, that row should be removed only if it is having duplicate and containtrue
at the same time. Allfalse
rows should be stay untouched in any case. That is, in this case, onlyAA, true
will be removed. But all answers leaves only one line -AA,false
. Just interesting :) – MiniMax Jul 21 '17 at 21:08