5

The duplicate is combination of different case text.

I need to count number of duplicate (case-insensitive) and then I need to remove duplicate by choose case with highest duplicate.

Below example:

hot chocolate
hot chocolate
hot chocolate
Hot Chocolate
Hot Chocolate
Hot Chocolate
Hot Chocolate
Hot Chocolate
Xicolatada
Xicolatada
Xicolatada
Xicolatada
XICOLATADA
XICOLATADA

Should become:

Hot Chocolate, 8
Xicolatada, 6

This question similar to this one but I need to choose case with highest duplicate and count case-insensitively.

  • Just curious, has anyone here ever needed to search for a string but only return whichever version of that string has the most instances based on case?!? this just seems like purely academic hoops that people are made to jump through in school and maybe would never, ever be needed in the real world! – Baazigar Jun 17 '15 at 18:21
6

And there's uniq --ignore-case --count | sort --numeric --reverse:

uniq -ic /tmp/foo.txt | sort -nr
      8 hot chocolate
      6 Xicolatada

And to switch around the order putting a comma in there add on:

... | sed -e 's/^ *\([0-9]*\) \(.*\)/\2, \1/'
5

I would use tolower() to make all the items lowercase. Then it is a matter of storing them in an array a[] and then printing the results:

$ awk '{a[tolower($0)]++} END {for (i in a) print i, a[i]}' file
xicolatada 6
hot chocolate 8

To have the output in comma-separated format, add -v OFS=,.

1

This will give you your desired output

use List::Util qw(sum);

my %count;
while (<>) {
    chomp;
    $count{+lc}{$_}++; 
}

$,=", ";
$\="\n";

while (my ($key, $hash) = each %count) {
    my @labels = reverse 
                 map  { $_->[0] }
                 sort { $a->[1] <=> $b->[1] } 
                 map  { [ $_, $hash->{$_} ] } 
                 keys %$hash;
    my $sum = sum values %$hash;

    print $labels[0], $sum;
}

Then

$ perl count.pl data.txt 
Hot Chocolate, 8
Xicolatada, 6

The order of the output is indeterminate.

  • You can use $b->[1] <=> $a->[1] to sort reverse, so we can drop reverse. – cuonglm Jun 17 '15 at 15:42
  • I find that less readable: the sort block already contains sufficient magic. – glenn jackman Jun 17 '15 at 16:32
0

If that list of items is in a file named list.txt, you can do:

tr '[:upper:]' '[:lower:]' < list.txt | sort | uniq -c

...which will output:

8 hot chocolate
6 xicolatada
0

POSIXly:

<in dd conv=lcase|LC_ALL=C sort|uniq -c >out

...which prints...

8 hot chocolate
6 xicolatada

Or with GNU tools:

<in LC_ALL=C sort -f|uniq -ic >out

...which prints...

8 Hot Chocolate
6 XICOLATADA

You need a GNU uniq there - or, well, you need one which supports the case -insensitive option, anyhow. All sorts should do that with -f anyway.

  • @cuonglm - is there any solution offered here which wouldn't need an explicit locale edited in? With the sample input given though, this one doesn't. – mikeserv Jun 17 '15 at 10:26
  • Yeah, all current answer need. The locale is for strictness. Feel free to revert it. – cuonglm Jun 17 '15 at 10:37
  • @cuonglm - nah. better is better. It's faster, anyway. So you know, uniq has the same problem. – mikeserv Jun 17 '15 at 10:38
  • 1
    GNU uniq does but POSIX uniq doesn't. GNU uniq -i using byte comparison instead of collation order so doesn't have. – cuonglm Jun 17 '15 at 10:41
  • @cuonglm - I upvoted that comment - but maybe in haste? I don't read that anywhere in the spec. It doesn't seem to specify anywhere how the lines should be compared, only that they should be. – mikeserv Jun 17 '15 at 10:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.