A file contains 5 columns with numbers
Example:
12 34 67 88 10
4 90 12 10 7
33 12 5 76 34
I would like to print the same number and see how many times it goes out. Example:
3 : 12
2 : 34
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this communityA file contains 5 columns with numbers
Example:
12 34 67 88 10
4 90 12 10 7
33 12 5 76 34
I would like to print the same number and see how many times it goes out. Example:
3 : 12
2 : 34
This awk
script prints output as in your example:
awk '{
for ( i=1; i<=NF; i++ ) # loop over all fields/columns
dict[$i]++; # count occurrence in an array using the field value as index/key
}
END { # after processing all data
for (key in dict) # iterate over all array keys
if(dict[key]>1) # if the key occurred more than once
print dict[key] " : " key # print counter and key
}' inputfile
With the example input, the output is
2 : 10
3 : 12
2 : 34
If you remove the condition if(a[i]>1)
it will also list numbers that occurred only once.
If you want to sort the result in descending order of the number of occurrence, append
| sort -nr
which means sort in reverse numerical order.
So the awk
command shown above combined with sort
awk '...' inputfile | sort -nr
produces
3 : 12
2 : 34
2 : 10
As mentioned in glenn jackman's comment you can instruct GNU AWK to sort the array values when processing with for
by adding PROCINFO["sorted_in"] = "@val_num_desc"
on top of the END
block.
END { # after processing all data
# In GNU AWK only you can use the next line to sort the array for processing
PROCINFO["sorted_in"] = "@val_num_desc" # sort descending by numeric value
for (key in dict) # iterate over all array keys
if(dict[key]>1) # if the key occurred more than once
print dict[key] " : " key # print counter and key
}
With this GNU specific extension you get sorted results without piping to sort
.
PROCINFO["sorted_in"] = "@val_num_desc"
at the top of the END block to sort the output: gnu.org/software/gawk/manual/html_node/…
Feb 22, 2019 at 20:12
You could use a pipeline
tr -s ' ' '\n' < datafile | sort | uniq -c -d
Depending on how refined you want your answer you could filter for numeric values. Remove the -d
to see all values, not just where the count is more than one.
| awk '($1 > 1) && ($2 > 0) { print $1 " : " $2 }'
to get output similar to the example in the question.
awk
rather than as a pipeline.
This is very similar to @roaima's answer, but the sed
lets us avoid having multiple spaces in the output when counting:
$ sed -E 's/ +/\n/g' file | sort | uniq -c -d
2 10
3 12
2 34
And, to sort numerically and add the :
, you can do:
$ sed -E 's/ +/\n/g' file | sort | uniq -c -d |
sort -rn | sed -E 's/([0-9]) /\1 : /'
3 : 12
2 : 34
2 : 10
Alternatively:
$ grep -oP '\d+' file | sort | uniq -c -d |
sort -rn | sed -E 's/([0-9]) /\1 : /'
3 : 12
2 : 34
2 : 10
Or, with perl
:
$ perl -lae '$k{$_}++ for @F;
END{
@keys = grep { $k{$_} > 1 } keys(%k);
@keys = sort { $k{$b} <=> $k{$a} } @keys;
print "$k{$_} : $_" for @keys
}' file
3 : 12
2 : 10
2 : 34
Or, if you're into the whole brevity thing:
$ perl -lae '$k{$_}++for@F}{print"$k{$_} : $_"for sort{$k{$b}<=>$k{$a}}grep{$k{$_}>1}keys(%k)' file
3 : 12
2 : 10
2 : 34
Assuming your input file is named bar
and is structured as nicely as you illustrate (whitespace and/or newlines between the numbers), one solution might be:
for n in $(cat bar); do echo "$n"; done | sort | uniq -c | sort -nr
command:
sed "N;s/\n/ /g" filename | sed "N;s/\n/ /g"| perl -pne "s/ /\n/g"| sed '/^$/d'| awk '{a[$1]++}END{for(x in a){print x,a[x]}}'|awk '$2 >1 {print $0}'
output
sed "N;s/\n/ /g" i.txt | sed "N;s/\n/ /g"| perl -pne "s/ /\n/g"| sed '/^$/d'| awk '{a[$1]++}END{for(x in a){print x,a[x]}}'|awk '$2 >1 {print $0}'
10 2
12 3
34 2
ok im resolved :
awk '($1 > 1) && ($2 > 0) { print $1 " : " $2 }' xxx.txt | sort | uniq -c | sort -nr
Thanks All
count : value
.