# Standard deviation of number next to a pattern

I have a file with lots of tags with a number next to them, set out like this:

``````<Score>4
... other data
<Score>2
... other data
<Score>3
``````

and so on...

I have used grep to get the number of times appears on a line

``````grep -c '<Score>' \$1
``````

and I have calculated the sum of the scores using awk:

``````awk 'sub(/<Score>/,""){y+=\$0} END{print y}' \$1}
``````

and then dividing it by the number I got from grep to get the mean.

What i'm struggling with is how to take away the mean from each value next to <Score>, squaring it and then summing them together.

The mean is stored in the variable \$mean The code I have tried looks like this:

``````awk 'sub(/<Score>/,""){y+=(\$0-\$mean)^2} END{print y}' \$1
``````

However it keeps outputting 0, if I can get this stored in a variable then I will be able to then calculate the Standard deviation, using all my other variables.

• Your mean is a shell variable, but it wont be expanded inside single quotes. Try passing it to awk using the extra args: `-v mean="\$mean"` and then using the awk variable this creates of the same name, but without a `\$` (...`y+=(\$0-mean)`...). – meuh Feb 21 '16 at 17:12
• Don't forget that the variance can be written as mean of square minus square of mean so (at the risk of some loss of floating point precision - not an issue if your data are integer, as shown in your example) you can make a single pass summing both `y+=\$0` and (say) `z+=\$0^2` and calculate both the mean and SD from those in your `END` block. – steeldriver Feb 21 '16 at 18:57

Here's a couple of utilities for you. The first one calculates the average of the numbers given to it (one number per line). The second one uses the first to calculate the standard deviation from the numbers in a file.

The executable file `average`:

``````#!/usr/bin/awk -f

/^[0-9.+-]/     { sum += \$0; ++n }

END             { print sum / n }
``````

This `awk` script will read input from a file or from standard input and compute the average of the numbers therein. It expects one number per line.

The executable file `stdev`:

``````#!/bin/sh

awk -v avg="\$( ./average "\$1" )" \
'/^[0-9.+-]/ { sum += (\$0 - avg)^2; ++n }
END         { print sqrt(sum / (n - 1)) }' "\$1"
``````

This shell script will first use the above `average` script to compute the average of the data in the file given on the command line. This number is assigned to the `awk` variable `avg`. It then uses the same kind of number detection as the `average` script to calculate the standard deviation.

As this script is written right now, it requires data from a file, not on standard input.

One way of using this on your data:

``````sed -n '/^<Score>/s///p' input.dat >output.dat
``````

With the given data, this will generate a file called `output.dat` containing the following:

``````4
2
3
``````

Using the `stdev` script above on this file:

``````\$ ./stdev output.dat
1
``````

Which is correct, as far as I can see.

Of course, you may do it directly in one `awk` call as well, without building any type of reusable tools:

``````awk -F '>' '/^<Score>/ { v[++n] = \$2; s += \$2 }
END { avg = s/n;
for (i=1; i<=n; ++i) {
std += (v[i] - avg)^2;
}
print sqrt(std / (n - 1));
}' input.dat
``````