I have a file with lots of tags with a number next to them, set out like this:

... other data
... other data

and so on...

I have used grep to get the number of times appears on a line

grep -c '<Score>' $1

and I have calculated the sum of the scores using awk:

awk 'sub(/<Score>/,""){y+=$0} END{print y}' $1}

and then dividing it by the number I got from grep to get the mean.

What i'm struggling with is how to take away the mean from each value next to <Score>, squaring it and then summing them together.

The mean is stored in the variable $mean The code I have tried looks like this:

awk 'sub(/<Score>/,""){y+=($0-$mean)^2} END{print y}' $1

However it keeps outputting 0, if I can get this stored in a variable then I will be able to then calculate the Standard deviation, using all my other variables.

  • 2
    Your mean is a shell variable, but it wont be expanded inside single quotes. Try passing it to awk using the extra args: -v mean="$mean" and then using the awk variable this creates of the same name, but without a $ (...y+=($0-mean)...). – meuh Feb 21 '16 at 17:12
  • 1
    Don't forget that the variance can be written as mean of square minus square of mean so (at the risk of some loss of floating point precision - not an issue if your data are integer, as shown in your example) you can make a single pass summing both y+=$0 and (say) z+=$0^2 and calculate both the mean and SD from those in your END block. – steeldriver Feb 21 '16 at 18:57

Here's a couple of utilities for you. The first one calculates the average of the numbers given to it (one number per line). The second one uses the first to calculate the standard deviation from the numbers in a file.

The executable file average:

#!/usr/bin/awk -f

/^[0-9.+-]/     { sum += $0; ++n }

END             { print sum / n }

This awk script will read input from a file or from standard input and compute the average of the numbers therein. It expects one number per line.

The executable file stdev:


awk -v avg="$( ./average "$1" )" \
    '/^[0-9.+-]/ { sum += ($0 - avg)^2; ++n }
     END         { print sqrt(sum / (n - 1)) }' "$1"

This shell script will first use the above average script to compute the average of the data in the file given on the command line. This number is assigned to the awk variable avg. It then uses the same kind of number detection as the average script to calculate the standard deviation.

As this script is written right now, it requires data from a file, not on standard input.

One way of using this on your data:

sed -n '/^<Score>/s///p' input.dat >output.dat

With the given data, this will generate a file called output.dat containing the following:


Using the stdev script above on this file:

$ ./stdev output.dat

Which is correct, as far as I can see.

Of course, you may do it directly in one awk call as well, without building any type of reusable tools:

awk -F '>' '/^<Score>/ { v[++n] = $2; s += $2 }
            END { avg = s/n;
                  for (i=1; i<=n; ++i) {
                      std += (v[i] - avg)^2;
                  print sqrt(std / (n - 1));
                 }' input.dat

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.