2

I have a large file that contains numeric data like this

123
124
124
124
126
127
127

I want to get total Number of repetitions (counted from each number occurring more than once). The output should be 5 as (124 is repeating 3 times and 127 two times). I am able to count repetitions using cat file | sort | uniq -d | wc -l but it gives output as 2 i.e two numbers are repeated (124 &127) and i want output 5.

5 Answers 5

8
awk 'seen[$0]++ {count += (seen[$0]==2 ? 2:1)} END {print count+0}' file

If a line has been seen before, increment count by either 2 or 1 (based on whether this line is the first duplicate). At the end print count (+0 so that awk prints 0 instead of an empty string in case count was never incremented).

Another approach:

awk '{count += seen[$0]; seen[$0] = (seen[$0]?1:2)} END {print count}' file

Instead of incrementing the seen array value, use the array to provide the amount by which count should be incremented – none the first time a line is seen, 2 for the second occurrence, then 1 for each subsequent duplicate.

6

Since you tagged your question linux you likely have the GNU implementation of uniq, which has a -D option:

   -D     print all duplicate lines

So

$ sort file | uniq -D | wc -l
5
2
  • Cool, didn't think about this ... Better than my answer and should be accepted :-)
    – pLumo
    Jul 27, 2020 at 10:23
  • @pLumo yours is more portable though Jul 27, 2020 at 10:25
5

You can use awk to count the numbers:

sort file | uniq -dc | awk '{n+=$1}END{print n}'

Output:

5

(you don't need cat here, as sort accepts input)

If your uniq does not support -dc, then

sort file | uniq -c | awk '$1>1{n+=$1}END{print n}'
4
  • Thanks. It's useful for me Jul 27, 2020 at 9:38
  • Unfortunately, uniq on macOS Catalina does not allow specifying both -c and -d at the same time.
    – Old Pro
    Jul 27, 2020 at 19:33
  • Then use only -c and add $1>1 to awk
    – pLumo
    Jul 27, 2020 at 19:36
  • @OldPro If you're using brew you can do brew search coreutils and then use guniq which accepts -c and -d
    – Matt
    Jul 29, 2020 at 11:26
3

With Perl:

perl -lne '
  $k += qw(2 1 0)[++$h{$_}<=>2];
  END { print $k; }
' file
5

We can compute the number of dups by maintaining a hash counter keyed on the input line.

The running counter $k is incremented in the amounts of 2, 1, and 0 when the key has been seen twice, more than two times, or the very first time.

Note the three -valued spaceship operator <=> which returns - 1,0,+1 upon comparison. See perldoc perlop for more clear details.

Note : strip whitespace(s) if any prior to to run this.

0

Tried with below method

awk '{a[$1]++}END{for(x in a){print x,a[x]}}' ppp| awk '$2 >1{sum=sum+$2}END{print sum}'

output

awk '{a[$1]++}END{for(x in a){print x,a[x]}}' ppp| awk '$2 >1{sum=sum+$2}END{print sum}'
5

Adding Python method

#!/usr/bin/python
m=open('ppp','r')
j=[]
f=[]
for i in m:
    if i.strip() not in j:
        j.append(i.strip())

e=open('ppp','r')
for i in e:
    f.append(i.strip())

r=0
for w in j:
    if f.count(w) >1:
        r=r+f.count(w)

print r


output
5

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .