Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a list of numbers in a file, one per line. How can I get the minimum, maximum, median and average values? I want to use the results in a bash script.

Although my immediate situation is for integers, a solution for floating-point numbers would be useful down the line, but a simple integer method is fine...

share|improve this question
2  
since you already know about bc, why can't you write it yourself? –  Kim May 25 '11 at 4:50
3  
I know about many things to some degree, but I am looking for a experienced person's view and methods... Isn't that was this site is all about? sharing knowledge... Yes I could probably work it out... and you could probably have worked out you SUID issue too... If my issue is anything like most questions here, I won't get the best solution on my own on my first attempt, and yet I may think it is... Hence my question... –  Peter.O May 25 '11 at 5:45
2  
yeah but your issue has been "worked out" a million times before and it seems you didn't even try to do it yourself. I always work on the issues myself before asking here and you should do the same, espescially if you want to gain knowledge. –  Kim May 25 '11 at 5:49
3  
You aren't getting it... It is trivial arithemetic to do it (add the numbers, divide, keep track of min, max ...mean).. The issue here is not "How?", it is "What is a good compact elegant way?". I've got rudimentary knowledge of awk (at best) , and I've used bc once for anything other than just looking at it, and I eventually asked a question after many hours (days actually) of chasing false trails. I got the answer on stackoverflow stackoverflow.com/questions/6007840/… –  Peter.O May 25 '11 at 6:23
add comment

8 Answers

up vote 18 down vote accepted

You can use the R programming language.

Suppose the data is in a file named datafile:

$ cat datafile
1
2
4

Invoke non-interactive R using Rscript:

$ Rscript -e 'd<-scan("datafile", quiet=TRUE); \
              cat(min(d), max(d), median(d), mean(d), sep="\n")'
1
4
2
2.333333

You can write an R script file:

#! /usr/bin/env Rscript
d<-scan("stdin", quiet=TRUE)
cat(min(d), max(d), median(d), mean(d), sep="\n")

Note the "stdin" which is a special filename to read from stdin.

And pipe the data to the R script:

$ cat datafile | ./mmmm.r
1
4
2
2.333333

Also works for floating points:

$ cat datafile
1.1
2.2
4.4
$ cat datafile | ./mmmm.r
1.1
4.4
2.2
2.566667

Read the fine R manuals at http://cran.r-project.org/manuals.html.

Unfortunately the full reference is only available in PDF. Another way to read the reference is by typing ?topicname in the prompt of an interactive R session.

share|improve this answer
1  
It looks interesting.. I'll have a closer look at it tomorrow.. Based on wikipedia's page, "R has become a de facto standard among statisticians"... well that's a significant accolade... I actaully tried to dowload it the other day (I kept seeing it mentioned), but I couldn't find it in the Ubuntu repo... I'll follow it up tomorrow... –  Peter.O May 25 '11 at 17:26
6  
in the ubuntu (and debian?) repo the package is named r-base. –  lesmana May 25 '11 at 17:44
    
thanks, I needed that name reference :) I didn't think of r- in the synaptic search field and it doesn't act on a lone character... I've tried it out now, and it looks ideal.. The R language is clearly the best for my requirement in this situation.. As per Gilles' answer, the Rscript interface to script files is most appropriate (vs. R, which is the interactive interface)... and R in the terminal makes for a handy calculator, or test environment (like python :) –  Peter.O May 26 '11 at 11:28
    
(+1) I love R. I can't recommend it enough. –  Dason Apr 3 '12 at 2:36
    
If you have data on stdin, you can use such a one-liner: { echo 'd<-scan()'; cat; echo; echo 'summary(d)'; } | R --slave –  Michał Wróbel Sep 2 '13 at 14:19
add comment

I actually keep a little awk program around to give the sum, data count, minimum datum, maximum datum, mean and median of a single column of numeric data:

#!/bin/sh
sort -n |
awk 'BEGIN{c=0;sum=0;}\
/^[^#]/{a[c++]=$1;sum+=$1;}\
END{ave=sum/c;\
if((c%2)==1){median=a[int(c/2)];}\
else{median=(a[c/2]+a[c/2-1])/2;}\
print sum,"\t",c,"\t",ave,"\t",median,"\t",a[0],"\t",a[c-1]}'

The above script reads from stdin, and prints tab-separated columns of output on a single line.

share|improve this answer
    
Aha! it's obvious (now that I've seen your awk script :) ... There is no need to keep checking for min and max when the array is sorted :) and that means that the NR==1 can go (a useless-use-of-if) along with the min/max checks, so all initializing can be located in the BEGIN section (good!)... Allowing for comments is a nice touch too.. Thanks, +1 ... –  Peter.O May 26 '11 at 2:28
    
Just a thought.. maybe allowing only numerics is better than disallowing comments (but that depends you your requirements).. –  Peter.O May 26 '11 at 6:21
add comment

Min, max and average are pretty easy to get with awk:

% echo -e '6\n2\n4\n3\n1' | awk 'NR == 1 { max=$1; min=$1; sum=0 }
   { if ($1>max) max=$1; if ($1<min) min=$1; sum+=$1;}
   END {printf "Min: %d\tMax: %d\tAverage: %f\n", min, max, sum/NR}'
Min: 1  Max: 6  Average: 3,200000

Calculating median is a bit more tricky, since you need to sort numbers and store them all in memory for a while or read them twice (first time to count them, second - to get median value). Here is example which stores all numbers in memory:

% echo -e '6\n2\n4\n3\n1' | sort -n | awk '{arr[NR]=$1}
   END { if (NR%2==1) print arr[(NR+1)/2]; else print (arr[NR/2]+arr[NR/2+1])/2}' 
3
share|improve this answer
    
Thanks... your example is a good lead-in to awk, for me.. I've tweaked it a bit and put the two together (getting the feel of awk)... I've used awk's asort rather than the piped sort, and it seems to sort integers and decimals correctly.. Here is a link to my resulting version paste.ubuntu.com/612674 ... (And a note to Kim: I've been experimenting with awk for a couple of hours now. Working with a personal-interest example is way better for me)... A general note to readers: I'm still interested to see other methods. the more compact the better. I'll wait a while ... –  Peter.O May 25 '11 at 11:06
add comment

I'll second lesmana's choice of R and offer my first R program. It reads one number per line on standard input and writes four numbers (min, max, average, median) separated by spaces to standard output.

#!/usr/bin/env Rscript
a <- scan(file("stdin"), c(0), quiet=TRUE);
cat(min(a), max(a), mean(a), median(a), "\n");
share|improve this answer
    
Thanks for the "second" (it's reassuring)... your example was useful, as I didn't realize straight-off that R is the interactive interface, and Rscript drives the scripted files, which can be executable as per your example hash-bang, or invoked from within a bash script.. The scripts can handle commandline args (eg. stackoverflow.com/questions/2045706/… ) so it's looking good... Also R expressions can be used in bash via the -e ... but I do wonder how R compares to bc ... –  Peter.O May 26 '11 at 2:05
add comment

Simple-r is the answer:

https://code.google.com/p/simple-r/

r summary file.txt
r -e 'min(d); max(d); median(d); mean(d)' file.txt

It uses R environment to simplify statistical analysis.

share|improve this answer
add comment

Just for the sake of having a variety of options presented on this page, Here are two more ways:

1: octave

  • GNU Octave is a high-level interpreted language, primarily intended for numerical computations. It provides capabilities for the numerical solution of linear and nonlinear problems, and for performing other numerical experiments.

Here is a quick octave example.

octave -q --eval 'A=1:10;
  printf ("# %f\t%f\t%f\t%f\n", min(A), max(A), median(A), mean(A));'  
# 1.000000        10.000000       5.500000        5.500000

2: bash + single-purpose tools.

For bash to handle floating-point numbers, this script uses numprocess and numaverage from package num-utils.

PS. I've also had a reasonable look at bc, but for this particular job, it doesn't offer anything beyond what awk does. It is (as the 'c' in 'bc' states) a calculator—a calculator which requires a much programming as awk and this bash script...


arr=($(sort -n "LIST" |tee >(numaverage 2>/dev/null >stats.avg) ))
cnt=${#arr[@]}; ((cnt==0)) && { echo -e "0\t0\t0\t0\t0"; exit; }
mid=$((cnt/2)); 
if [[ ${cnt#${cnt%?}} == [02468] ]] 
   then med=$( echo -n "${arr[mid-1]}" |numprocess /+${arr[mid]},%2/ )
   else med=${arr[mid]}; 
fi     #  count   min       max           median        average
echo -ne "$cnt\t${arr[0]}\t${arr[cnt-1]}\t$med\t"; cat stats.avg 
share|improve this answer
add comment
nums=$(<file.txt); 
list=(`for n in $nums; do printf "%015.06f\n" $n; done | sort -n`); 
echo min ${list[0]}; 
echo max ${list[${#list[*]}-1]}; 
echo median ${list[${#list[*]}/2]};
share|improve this answer
    
echo file.txt does not looks quite right, maybe cat –  malat Dec 17 '13 at 14:14
add comment

If you're more interested in utility rather than being cool or clever, then perl is an easier choice than awk. By and large it will be on every *nix with consistent behaviour, and is easy and free to install on windows. I think it's also less cryptic than awk, and there will be some stats modules you could use if you wanted a halfway house between writing it yourself and something like R. My fairly untested (in fact I know it has bugs but it works for my purposes) perl script took about a minute to write, and I'd guess the only cryptic part would be the while(<>), which is the very useful shorthand, meaning take the file(s) passed as command line arguments, read a line at a time and put that line in the special variable $_. So you could put this in a file called count.pl and run it as perl count.pl myfile. Apart from that it should be painfully obvious what's going on.

$max = 0;
while (<>) {
 $sum = $sum + $_;
 $max = $_ if ($_ > $max);
 $count++;
}
$avg=$sum/$count;
print "$count numbers total=$sum max=$max mean=$avg\n";
share|improve this answer
2  
You haven't shown the median –  Peter.O Mar 28 '12 at 14:31
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.