1

I have 3000 files (1.out, 2.out, 3.out...., each with a single column something like this:

0.446477
0.439331
0.444394
0.425003
0.428981
0.419547
0.432834
0.417874
........

I need to calculate average and standard deviation for each row across 3000 files. I could calculate the average using:

awk '{a[FNR]+=$1;b[FNR]++;}END{for(i=1;i<=FNR;i++)print a[i]/b[i];}' *.out

But I am stuck with the calculation of standard deviation.

  • Where are you stuck? You don't know the formula for standard deviation? – muru Aug 7 '17 at 4:42
  • @muru The formula is sqrt (1/n*(sum(Xi-mean)^2)). I don't know how to use this in scripting. Thanks ! – ashu Aug 7 '17 at 4:50
  • You can use sqrt in awk: unix.stackexchange.com/a/336613/70524 – muru Aug 7 '17 at 4:56
  • do you want to print average and standard deviation for each file separately? – RomanPerekhrest Aug 7 '17 at 5:36
  • @RomanPerekhrest : I need a single average i.e. First row from each file to be averaged and written as single line in out file. Followed by second line from each file..like wise. And each row will have a standard deviation value for it's average. Thanks ! – ashu Aug 7 '17 at 5:47
2

Complex solution using paste command and datamash tool:

  1. Merging all files into one file for further processing:

    paste [0-9]*.out | datamash transpose > data
    

    -- datamash transpose- will transpose rows into columns

  2. Getting total number of fields:

    nf=`awk '{print NF; exit}' data`
    
  3. Calculating mean and standard deviation:

    for ((i=1; i<$nf; i++)); do datamash mean $i pstdev $i < data; done 
    

The output would look like (1st column - mean value, 2nd column - st. deviation value):

0.596477    0.11180339887499
0.589331    0.11180339887499
0.594394    0.11180339887499
0.575003    0.11180339887499
0.578981    0.11180339887499
0.569547    0.11180339887499
0.582834    0.11180339887499
.......
  • Thanks a lot ! Though looks like datamash is not installed on the Redhat work station I am working. I need to talk to my sys admin for this. – ashu Aug 7 '17 at 8:41
  • @ashu, welcome, datamash is indeed a good tool for such cases – RomanPerekhrest Aug 7 '17 at 8:56
1

Standard deviation can be calculated in one pass without modifying your script much.

awk '{a[FNR]+=$1; b[FNR]++; c[FNR]+=$1*$1 } 
     END{
        for(i=1;i<=FNR;i++)
          print a[i]/b[i], sqrt((c[i]-a[i]*a[i]/b[i])/(b[i]-1)) ;
     }
' *.out

This is the naive implementation, another implementation, the "online method" on the same wiki page :

awk '{
  x=$1
  n[FNR] += 1
  delta = x - mean[FNR]
  mean[FNR] += delta/n[FNR]
  delta2 = x - mean[FNR]
  M2[FNR] += delta * delta2
}
END{
for(i=1;i<=FNR;i++)
        if(n[i]<2)
                print mean[i], 0
        else
                print mean[i], sqrt(M2[i]/(n[i]-1))
}' *.out

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.