2

A file contains 5 columns with numbers

Example:

12 34 67 88 10
 4 90 12 10 7
33 12 5  76 34

I would like to print the same number and see how many times it goes out. Example:

3 : 12
2 : 34
  • presumably only for numbers that occur more than once? – Jeff Schaller Feb 22 at 15:55
  • @Kusalananda it's count : value. – roaima Feb 22 at 18:42
  • Are you asking about finding and counting duplicates? Is that what you mean by "print the same number"? That's very unclear because you don't say what it has to be the same as. – Peter Cordes Feb 22 at 23:16
  • I suppose you want the count of how many times each number appears, so edited the title to actually say that. – ilkkachu Feb 24 at 15:06
9

You could use a pipeline

tr -s ' ' '\n' < datafile | sort | uniq -c -d

Depending on how refined you want your answer you could filter for numeric values. Remove the -d to see all values, not just where the count is more than one.

  • You could append | awk '($1 > 1) && ($2 > 0) { print $1 " : " $2 }' to get output similar to the example in the question. – Bodo Feb 22 at 16:33
  • This is OK but print not in order increase – InsideMiamiTattoo Feb 22 at 16:54
  • @InsideMiamiTattoo where in your question did you say anything about a sorted output? Do you want to sort by count or by value? – roaima Feb 22 at 17:08
  • @Bodo agreed, but if I did that I would have addressed the entire request with awk rather than as a pipeline. – roaima Feb 22 at 17:09
  • 1
    maybe use tr -s to avoid blank lines due to multiple spaces. – glenn jackman Feb 22 at 20:10
9

This awk script prints output as in your example:

awk '{ 
         for ( i=1; i<=NF; i++ ) # loop over all fields/columns
            dict[$i]++;      # count occurrence in an array using the field value as index/key
     }
 END {                           # after processing all data
         for (key in dict)       # iterate over all array keys
             if(dict[key]>1)     # if the key occurred more than once
                 print dict[key] " : " key    # print counter and key
     }' inputfile

With the example input, the output is

2 : 10
3 : 12
2 : 34

If you remove the condition if(a[i]>1) it will also list numbers that occurred only once.

If you want to sort the result in descending order of the number of occurrence, append

| sort -nr

which means sort in reverse numerical order.

So the awk command shown above combined with sort

awk '...' inputfile | sort -nr

produces

3 : 12
2 : 34
2 : 10

As mentioned in glenn jackman's comment you can instruct GNU AWK to sort the array values when processing with for by adding PROCINFO["sorted_in"] = "@val_num_desc" on top of the END block.

 END {                           # after processing all data
         # In GNU AWK only you can use the next line to sort the array for processing
         PROCINFO["sorted_in"] = "@val_num_desc" # sort descending by numeric value
         for (key in dict)       # iterate over all array keys
             if(dict[key]>1)     # if the key occurred more than once
                 print dict[key] " : " key    # print counter and key
     }

With this GNU specific extension you get sorted results without piping to sort.

2

This is very similar to @roaima's answer, but the sed lets us avoid having multiple spaces in the output when counting:

$ sed -E 's/ +/\n/g' file | sort | uniq -c -d
      2 10
      3 12
      2 34

And, to sort numerically and add the :, you can do:

$ sed -E 's/ +/\n/g' file | sort | uniq -c -d | 
    sort -rn | sed -E 's/([0-9]) /\1 : /'
      3 : 12
      2 : 34
      2 : 10

Alternatively:

$ grep -oP '\d+' file | sort | uniq -c -d | 
    sort -rn | sed -E 's/([0-9]) /\1 : /'
      3 : 12
      2 : 34
      2 : 10

Or, with perl:

$ perl -lae '$k{$_}++ for @F; 
              END{ 
                @keys = grep { $k{$_} > 1 } keys(%k);  
                @keys = sort { $k{$b} <=> $k{$a} } @keys;

                print "$k{$_} : $_" for @keys
              }' file
3 : 12
2 : 10
2 : 34

Or, if you're into the whole brevity thing:

$ perl -lae '$k{$_}++for@F}{print"$k{$_} : $_"for sort{$k{$b}<=>$k{$a}}grep{$k{$_}>1}keys(%k)' file 
3 : 12
2 : 10
2 : 34
1

Assuming your input file is named bar and is structured as nicely as you illustrate (whitespace and/or newlines between the numbers), one solution might be:

for n in $(cat bar); do echo "$n"; done | sort | uniq -c | sort -nr
0

command:

sed "N;s/\n/ /g" filename | sed "N;s/\n/ /g"| perl -pne "s/ /\n/g"| sed '/^$/d'| awk '{a[$1]++}END{for(x in a){print x,a[x]}}'|awk '$2 >1 {print $0}'

output

sed "N;s/\n/ /g" i.txt | sed "N;s/\n/ /g"| perl -pne "s/ /\n/g"| sed '/^$/d'| awk '{a[$1]++}END{for(x in a){print x,a[x]}}'|awk '$2 >1 {print $0}'

10 2
12 3
34 2
-3

ok im resolved :

awk '($1 > 1) && ($2 > 0) { print $1 " : " $2 }' xxx.txt | sort | uniq -c | sort -nr

Thanks All

  • 2
    That doesn't seem to address your requirement at all. Have you tested it? – roaima Feb 22 at 18:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.