2

I have 200 text files. Each text file contains one column of mathematical numbers (100k rows).

I want to do some statistical processing on these data (finding the probit or quantile). I know the mean and sigma value associated for each file. Is there any direct way to calculate probit or quantile of a dataset?

If not, then I know the formula for quantile which is =(sorted data-mean)/sigma. e.g.

input_file.txt

12.2  
10.11  
15.55  
5.01  
...
...
12.9

temp.txt

5.01  
10.11  
12.2  
12.9  
15.55  
... 
...

Final output file should contain two columns. First column is same as temp_output.txt and second column is the following mathematical function on each element:

= (row(i)-mean)/sigma

Where mean and sigma are two mathematical number say 10.1 and 2.02 respectively. Final output file should look like this for above values

output.txt

5.01 -2.51980198  
10.11 0.004950495  
12.2 1.03960396  
12.9 1.386138614  
15.55 2.698019802  
... 
...
  • 3
    If someone does you the courtesy of editing your question to make it (a) more readable, and (b) formatted to the U&L style guidelines, please don't roll it back: it is both rude and counter productive. – jasonwryan Sep 12 '14 at 5:35
  • Sorry, that was not the intention. I want to edit one word in the question to make it more clear. Thanks for editing and formatting. – Nilesh Sep 12 '14 at 5:49
  • That's cool; it's a wiki, it is suposed to be edited. I just didn't understand why you completely removed my changes. – jasonwryan Sep 12 '14 at 5:51
  • 2
    @jasonwryan This happens with simultaneous edits sometimes. – Bernhard Sep 12 '14 at 5:55
  • 3
    Nilesh, please install R, and look over at Stack Overflow for recipes. – Deer Hunter Sep 12 '14 at 6:12
2

do you meed something like this

mybox $ cat file1.txt
12.2
10.11
15.55
5.01
mybox $ awk -v mean=10.2 -v sigma=2.02 '{printf "%f %f\n",$1,($1-mean)/sigma ; }' file1.txt
12.200000 0.990099
10.110000 -0.044554
15.550000 2.648515
5.010000 -2.569307

the cryptic part

awk -v mean=10.2 -v sigma=2.02 '{printf "%f %f\n",$1,($1-mean)/sigma ; }' file1.txt

how do you intend to provide mean1,mean2, ... mean200 ?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.