count how many times each number occurs

Question

A file contains 5 columns with numbers

Example:

12 34 67 88 10
 4 90 12 10 7
33 12 5  76 34

I would like to print the same number and see how many times it goes out. Example:

3 : 12
2 : 34

Answer 1

This awk script prints output as in your example:

awk '{ 
         for ( i=1; i<=NF; i++ ) # loop over all fields/columns
            dict[$i]++;      # count occurrence in an array using the field value as index/key
     }
 END {                           # after processing all data
         for (key in dict)       # iterate over all array keys
             if(dict[key]>1)     # if the key occurred more than once
                 print dict[key] " : " key    # print counter and key
     }' inputfile

With the example input, the output is

2 : 10
3 : 12
2 : 34

If you remove the condition if(a[i]>1) it will also list numbers that occurred only once.

If you want to sort the result in descending order of the number of occurrence, append

| sort -nr

which means sort in reverse numerical order.

So the awk command shown above combined with sort

awk '...' inputfile | sort -nr

produces

3 : 12
2 : 34
2 : 10

As mentioned in glenn jackman's comment you can instruct GNU AWK to sort the array values when processing with for by adding PROCINFO["sorted_in"] = "@val_num_desc" on top of the END block.

 END {                           # after processing all data
         # In GNU AWK only you can use the next line to sort the array for processing
         PROCINFO["sorted_in"] = "@val_num_desc" # sort descending by numeric value
         for (key in dict)       # iterate over all array keys
             if(dict[key]>1)     # if the key occurred more than once
                 print dict[key] " : " key    # print counter and key
     }

With this GNU specific extension you get sorted results without piping to sort.

Answer 2

You could use a pipeline

tr -s ' ' '\n' < datafile | sort | uniq -c -d

Depending on how refined you want your answer you could filter for numeric values. Remove the -d to see all values, not just where the count is more than one.

Answer 3

This is very similar to @roaima's answer, but the sed lets us avoid having multiple spaces in the output when counting:

$ sed -E 's/ +/\n/g' file | sort | uniq -c -d
      2 10
      3 12
      2 34

And, to sort numerically and add the :, you can do:

$ sed -E 's/ +/\n/g' file | sort | uniq -c -d | 
    sort -rn | sed -E 's/([0-9]) /\1 : /'
      3 : 12
      2 : 34
      2 : 10

Alternatively:

$ grep -oP '\d+' file | sort | uniq -c -d | 
    sort -rn | sed -E 's/([0-9]) /\1 : /'
      3 : 12
      2 : 34
      2 : 10

Or, with perl:

$ perl -lae '$k{$_}++ for @F; 
              END{ 
                @keys = grep { $k{$_} > 1 } keys(%k);  
                @keys = sort { $k{$b} <=> $k{$a} } @keys;

                print "$k{$_} : $_" for @keys
              }' file
3 : 12
2 : 10
2 : 34

Or, if you're into the whole brevity thing:

$ perl -lae '$k{$_}++for@F}{print"$k{$_} : $_"for sort{$k{$b}<=>$k{$a}}grep{$k{$_}>1}keys(%k)' file 
3 : 12
2 : 10
2 : 34

Answer 4

Assuming your input file is named bar and is structured as nicely as you illustrate (whitespace and/or newlines between the numbers), one solution might be:

for n in $(cat bar); do echo "$n"; done | sort | uniq -c | sort -nr

Answer 5

command:

sed "N;s/\n/ /g" filename | sed "N;s/\n/ /g"| perl -pne "s/ /\n/g"| sed '/^$/d'| awk '{a[$1]++}END{for(x in a){print x,a[x]}}'|awk '$2 >1 {print $0}'

output

sed "N;s/\n/ /g" i.txt | sed "N;s/\n/ /g"| perl -pne "s/ /\n/g"| sed '/^$/d'| awk '{a[$1]++}END{for(x in a){print x,a[x]}}'|awk '$2 >1 {print $0}'

10 2
12 3
34 2

Answer 6

ok im resolved :

awk '($1 > 1) && ($2 > 0) { print $1 " : " $2 }' xxx.txt | sort | uniq -c | sort -nr

Thanks All