# How to count the number of occurrences of all strings in the file while getting the average of each string

I am trying to count the number of occurrences without specifying a specific string, just using the output of the cut command to print the number of each repeated string using grep. I then want to get the average using numaverage, but am unsure how to accomplish this without first removing the numbers.
I first used the command `cut -d " " -f 1 \$file` to half the file to see only the left side ex:

``````NEUTRON   20.900103
PION-      0.215176
PION-     22.716532
NEUTRON    8.043279
PION+      1.374297
PION-      0.313350
PION+      0.167848
``````

to then just

``````NEUTRON
PION-
PION-
NEUTRON
PION+
PION-
PION+
``````

How could I | together cut,grep, numaverage in this mannor(other commands may help such as cat, uniq, wc)? ex output:

``````Name          count     Average
KAON-            1      5.489958
NEUTRON          2      14.471691
PHOTON          10      0.652727
PION-            5      5.145192
PION+            7      2.691639
PROTON           1      1.160216
``````
• How is average calculated? – Shawn Oct 11 '19 at 3:55
• And it helps if your expected output is derived from the sample input instead of being apparently unrelated. Half the terms in the output aren't even in the input and the counts of the others don't match. – Shawn Oct 11 '19 at 3:58
• Use python or perl or awk to increment a counter indexed by the first field and a sum, also indexed; then at the end, output count and average. Ask back if you should encounter any problems. – Philippos Oct 11 '19 at 4:36

Try `awk`:

``````awk '
BEGIN{FS=OFS="\t"} # if your file is space-delimited, leave this out
{c[\$1]++;v[\$1]+=\$2}
END{
print "Name","count","Average"
for(f in c){print f,c[f],v[f]/c[f]}
}
' file
``````

Use `csvsql` from `csvkit`:

``````csvsql -d' ' -S -H --query 'select a as Name, count(*) as count, avg(b) as Average from file group by a' file \
| csvformat -D' '
``````

or if your file is tab-delimited:

``````csvsql -t -S -H --query 'select a as Name, count(*) as count, avg(b) as Average from file group by a' file \
| csvformat -T
``````

Output:

``````Name    count   Average
NEUTRON 2   14.471691
PION+   2   0.7710725
PION-   3   7.74835266667
``````

You might need to install it with `pip`:

``````pip install csvkit
``````

I like both answers (SQL, AWK) because they are standard languages and implement the algorithm in a idiomatic way. I was about to downvote the Q even further though, because it seems to insist on a specific solution.

But that exactly is fgdark's problem: you rely too much on utilities and do not understand the algorithm(s) involved.

How could I pipe together cut, grep, numaverage in this manor (other commands may help such as cat, uniq, wc)?

You can "easily" find a pipe solution by starting with temporary files, and when the chain works, you pipe it together. The result will be unsatisfactory. I don't have numaverage, for starters. Performance also.

Here, some "real" programming language is needed, like SQL or AWK. Or perl or python or C. Or bash - but bash as a script language with associative arrays and arithmetic operators, not bash "only" as a shell.

You don't need a average() function as with the SQL solution, you can just count, add and divide the data you have read in to your arrays.

So if you want to do more data reporting, you should pick a language. SQL is a bit special, being strong on data mining. The others mentioned (awk, perl, python, C, even bash) are strong in data processing (manipulating variables/arrays/structures). But there is large overlap.

I worked with SQL for some years; this is how I would save the query and it's call:

``````SQL='select a as Name, count(*) as Count, avg(b) as Average'
SQL+=' from file'
SQL+=' group by a'

csvsql -tSH --query \$SQL file | csvformat -T
``````

'Tis not a critique, but an attempt to illustrate how the Structured Query Language is - structured. It is a textbook demonstration of a `group by` clause with aggregate functions `count` and `avg`.

Bash provides the syntax `+=` to easily do that.