Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a list of numbers in a file, one per line. How can I get the minimum, maximum, median and average values? I want to use the results in a bash script.

Although my immediate situation is for integers, a solution for floating-point numbers would be useful down the line, but a simple integer method is fine...

share|improve this question

16 Answers 16

up vote 35 down vote accepted

You can use the R programming language.

Here is a quick and dirty R script:

#! /usr/bin/env Rscript
d<-scan("stdin", quiet=TRUE)
cat(min(d), max(d), median(d), mean(d), sep="\n")

Note the "stdin" in scan which is a special filename to read from standard input (that means from pipes or redirections).

Now you can redirect your data over stdin to the R script:

$ cat datafile
1
2
4
$ ./mmmm.r < datafile
1
4
2
2.333333

Also works for floating points:

$ cat datafile2
1.1
2.2
4.4
$ ./mmmm.r < datafile2
1.1
4.4
2.2
2.566667

If you don't want to write an R script file you can invoke a true one-liner (with linebreak only for readability) in the command line using Rscript:

$ Rscript -e 'd<-scan("stdin", quiet=TRUE)' \
          -e 'cat(min(d), max(d), median(d), mean(d), sep="\n")' < datafile
1
4
2
2.333333

Read the fine R manuals at http://cran.r-project.org/manuals.html.

Unfortunately the full reference is only available in PDF. Another way to read the reference is by typing ?topicname in the prompt of an interactive R session.


For completeness: there is an R command which outputs all the values you want and more. Unfortunately in a human friendly format which is hard to parse programmatically.

> summary(c(1,2,4))
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  1.000   1.500   2.000   2.333   3.000   4.000 
share|improve this answer
1  
It looks interesting.. I'll have a closer look at it tomorrow.. Based on wikipedia's page, "R has become a de facto standard among statisticians"... well that's a significant accolade... I actaully tried to dowload it the other day (I kept seeing it mentioned), but I couldn't find it in the Ubuntu repo... I'll follow it up tomorrow... – Peter.O May 25 '11 at 17:26
7  
in the ubuntu (and debian?) repo the package is named r-base. – lesmana May 25 '11 at 17:44
    
thanks, I needed that name reference :) I didn't think of r- in the synaptic search field and it doesn't act on a lone character... I've tried it out now, and it looks ideal.. The R language is clearly the best for my requirement in this situation.. As per Gilles' answer, the Rscript interface to script files is most appropriate (vs. R, which is the interactive interface)... and R in the terminal makes for a handy calculator, or test environment (like python :) – Peter.O May 26 '11 at 11:28
    
(+1) I love R. I can't recommend it enough. – Dason Apr 3 '12 at 2:36
4  
or just cat datafile | Rscript -e 'print(summary(scan("stdin")));' – shabbychef Aug 11 '14 at 22:32

I actually keep a little awk program around to give the sum, data count, minimum datum, maximum datum, mean and median of a single column of numeric data:

#!/bin/sh
sort -n | awk '
  BEGIN {
    c = 0;
    sum = 0;
  }
  $1 ~ /^[0-9]*(\.[0-9]*)?$/ {
    a[c++] = $1;
    sum += $1;
  }
  END {
    ave = sum / c;
    if( (c % 2) == 1 ) {
      median = a[ int(c/2) ];
    } else {
      median = ( a[c/2] + a[c/2-1] ) / 2;
    }
    OFS="\t";
    print sum, c, ave, median, a[0], a[c-1];
  }
'

The above script reads from stdin, and prints tab-separated columns of output on a single line.

share|improve this answer
1  
Aha! it's obvious (now that I've seen your awk script :) ... There is no need to keep checking for min and max when the array is sorted :) and that means that the NR==1 can go (a useless-use-of-if) along with the min/max checks, so all initializing can be located in the BEGIN section (good!)... Allowing for comments is a nice touch too.. Thanks, +1 ... – Peter.O May 26 '11 at 2:28
    
Just a thought.. maybe allowing only numerics is better than disallowing comments (but that depends you your requirements).. – Peter.O May 26 '11 at 6:21
    
Technically, awk will assume "new" variables are zero, so in this case the BEGIN{} section is unnecessary. I've fixed the wrapping (no need to escape the line breaks either). I also used OFS="\t" to clean up the print line and implemented @Peter.O's second comment. (Yes, my regex allows ., but as awk interprets that as 0, that's acceptable.) – Adam Katz Jan 15 '15 at 21:22
    
@AdamKatz - these are great changes, but as it stands, I didn't write the program. My awk script is now substantially different. I almost feel like you should take credit for the above program, in order to give credit where credit is due. – Bruce Ediger Jan 15 '15 at 22:31

Min, max and average are pretty easy to get with awk:

% echo -e '6\n2\n4\n3\n1' | awk 'NR == 1 { max=$1; min=$1; sum=0 }
   { if ($1>max) max=$1; if ($1<min) min=$1; sum+=$1;}
   END {printf "Min: %d\tMax: %d\tAverage: %f\n", min, max, sum/NR}'
Min: 1  Max: 6  Average: 3,200000

Calculating median is a bit more tricky, since you need to sort numbers and store them all in memory for a while or read them twice (first time to count them, second - to get median value). Here is example which stores all numbers in memory:

% echo -e '6\n2\n4\n3\n1' | sort -n | awk '{arr[NR]=$1}
   END { if (NR%2==1) print arr[(NR+1)/2]; else print (arr[NR/2]+arr[NR/2+1])/2}' 
3
share|improve this answer
    
Thanks... your example is a good lead-in to awk, for me.. I've tweaked it a bit and put the two together (getting the feel of awk)... I've used awk's asort rather than the piped sort, and it seems to sort integers and decimals correctly.. Here is a link to my resulting version paste.ubuntu.com/612674 ... (And a note to Kim: I've been experimenting with awk for a couple of hours now. Working with a personal-interest example is way better for me)... A general note to readers: I'm still interested to see other methods. the more compact the better. I'll wait a while ... – Peter.O May 25 '11 at 11:06

pythonpy works well for this sort of thing:

cat file.txt | py --ji -l 'min(l), max(l), numpy.median(l), numpy.mean(l)'
share|improve this answer

With GNU datamash:

$ printf '1\n2\n4\n' | datamash max 1 min 1 mean 1 median 1
4   1   2.3333333333333 2
share|improve this answer
nums=$(<file.txt); 
list=(`for n in $nums; do printf "%015.06f\n" $n; done | sort -n`); 
echo min ${list[0]}; 
echo max ${list[${#list[*]}-1]}; 
echo median ${list[${#list[*]}/2]};
share|improve this answer
    
echo file.txt does not looks quite right, maybe cat – malat Dec 17 '13 at 14:14

And a Perl one-(long)liner, including median:

cat numbers.txt \
| perl -M'List::Util qw(sum max min)' -MPOSIX -0777 -a -ne 'printf "%-7s : %d\n"x4, "Min", min(@F), "Max", max(@F), "Average", sum(@F)/@F,  "Median", sum( (sort {$a<=>$b} @F)[ int( $#F/2 ), ceil( $#F/2 ) ] )/2;'

The special options used are:

  • -0777 : read the whole file at once instead of line by line
  • -a : autosplit into the @F array

A more readable script version of the same thing would be :

#!/usr/bin/perl

use List::Util qw(sum max min);
use POSIX;

@F=<>;

printf "%-7s : %d\n" x 4,
    "Min", min(@F),
    "Max", max(@F),
    "Average", sum(@F)/@F,
    "Median", sum( (sort {$a<=>$b} @F)[ int( $#F/2 ), ceil( $#F/2 ) ] )/2;

If you want decimals, replace %d with something like %.2f.

share|improve this answer

Simple-r is the answer:

r summary file.txt
r -e 'min(d); max(d); median(d); mean(d)' file.txt

It uses R environment to simplify statistical analysis.

share|improve this answer

I'll second lesmana's choice of R and offer my first R program. It reads one number per line on standard input and writes four numbers (min, max, average, median) separated by spaces to standard output.

#!/usr/bin/env Rscript
a <- scan(file("stdin"), c(0), quiet=TRUE);
cat(min(a), max(a), mean(a), median(a), "\n");
share|improve this answer
    
Thanks for the "second" (it's reassuring)... your example was useful, as I didn't realize straight-off that R is the interactive interface, and Rscript drives the scripted files, which can be executable as per your example hash-bang, or invoked from within a bash script.. The scripts can handle commandline args (eg. stackoverflow.com/questions/2045706/… ) so it's looking good... Also R expressions can be used in bash via the -e ... but I do wonder how R compares to bc ... – Peter.O May 26 '11 at 2:05

Just for the sake of having a variety of options presented on this page, Here are two more ways:

1: octave

  • GNU Octave is a high-level interpreted language, primarily intended for numerical computations. It provides capabilities for the numerical solution of linear and nonlinear problems, and for performing other numerical experiments.

Here is a quick octave example.

octave -q --eval 'A=1:10;
  printf ("# %f\t%f\t%f\t%f\n", min(A), max(A), median(A), mean(A));'  
# 1.000000        10.000000       5.500000        5.500000

2: bash + single-purpose tools.

For bash to handle floating-point numbers, this script uses numprocess and numaverage from package num-utils.

PS. I've also had a reasonable look at bc, but for this particular job, it doesn't offer anything beyond what awk does. It is (as the 'c' in 'bc' states) a calculator—a calculator which requires a much programming as awk and this bash script...


arr=($(sort -n "LIST" |tee >(numaverage 2>/dev/null >stats.avg) ))
cnt=${#arr[@]}; ((cnt==0)) && { echo -e "0\t0\t0\t0\t0"; exit; }
mid=$((cnt/2)); 
if [[ ${cnt#${cnt%?}} == [02468] ]] 
   then med=$( echo -n "${arr[mid-1]}" |numprocess /+${arr[mid]},%2/ )
   else med=${arr[mid]}; 
fi     #  count   min       max           median        average
echo -ne "$cnt\t${arr[0]}\t${arr[cnt-1]}\t$med\t"; cat stats.avg 
share|improve this answer

Minimum:

jq -s min

Maximum:

jq -s max

Median:

sort -n|awk '{a[NR]=$0}END{if(NR%2==1)print a[int(NR/2)+1];else print(a[NR/2-1]+a[NR/2])/2}'

Average:

jq -s 'add/length'

-s (--slurp) reads input lines into an array in jq.

share|improve this answer

Taking cues from Bruce's code, here is a more efficient implementation which does not keep the whole data in memory.  As stated in the question, it assumes that the input file has (at most) one number per line.  It counts the lines in the input file that contain a qualifying number and passes the count to the awk command along with (preceding) the sorted data.  So, for example, if the file contains

6.0
4.2
8.3
9.5
1.7

then the input to awk is actually

5
1.7
4.2
6.0
8.3
9.5

Then the awk script captures the data count in the NR==1 code block and saves the middle value (or the two middle values, which are averaged to yield the median) when it sees them.

FILENAME="Salaries.csv"

(awk 'BEGIN {c=0} $1 ~ /^[-0-9]*(\.[0-9]*)?$/ {c=c+1;} END {print c;}' "$FILENAME"; \
        sort -n "$FILENAME") | awk '
  BEGIN {
    c = 0
    sum = 0
    med1_loc = 0
    med2_loc = 0
    med1_val = 0
    med2_val = 0
    min = 0
    max = 0
  }

  NR==1 {
    LINES = $1
    # We check whether numlines is even or odd so that we keep only
    # the locations in the array where the median might be.
    if (LINES%2==0) {med1_loc = LINES/2-1; med2_loc = med1_loc+1;}
    if (LINES%2!=0) {med1_loc = med2_loc = (LINES-1)/2;}
  }

  $1 ~ /^[-0-9]*(\.[0-9]*)?$/  &&  NR!=1 {
    # setting min value
    if (c==0) {min = $1;}
    # middle two values in array
    if (c==med1_loc) {med1_val = $1;}
    if (c==med2_loc) {med2_val = $1;}
    c++
    sum += $1
    max = $1
  }
  END {
    ave = sum / c
    median = (med1_val + med2_val ) / 2
    print "sum:" sum
    print "count:" c
    print "mean:" ave
    print "median:" median
    print "min:" min
    print "max:" max
  }
'
share|improve this answer
    
Welcome to Unix & Linux!  Good job for a first post.  (1) While this may answer the question, it would be a better answer if you could explain how/why it does so.  The site’s standards have evolved over the past four years; while code-only answers were acceptable in 2011, we now prefer comprehensive answers that provide more explanation and context.  I’m not asking you to explain the entire script; just the parts that you changed (but if you want to explain the entire script, that’s OK too).  (BTW, I understand it fine; I’m asking on behalf of our less experienced users.)  … (Cont’d) – G-Man Oct 10 '15 at 6:18
    
(Cont’d) …  Please do not respond in comments; edit your answer to make it clearer and more complete.  (2) Fixing the script so that it does not need to hold the entire array in memory is a good improvement, but I’m not sure whether it’s appropriate to say that your version is “more efficient” when you have three unnecessary cat commands; see UUOC.  … (Cont’d) – G-Man Oct 10 '15 at 6:19
    
(Cont’d) …  (3) Your code is safe, since you set FILENAME and you know what you set it to, but, in general, you should always quote shell variables unless you have a good reason not to, and you’re sure you know what you’re doing.  (4) Both your answer and Bruce’s ignore negative input (i.e., numbers beginning with -); there is nothing in the question to suggest that this is correct or desired behavior.  Don’t feel bad; it’s been over four years, and, apparently, I’m the first person who noticed. – G-Man Oct 10 '15 at 6:20
    
Made edits as per suggestions. Didn,t knew about the overhead of cat command. Always used it to stream single files. Thanks for telling me about UUOC..... – Rahul Agarwal Oct 10 '15 at 15:40
    
Good.  I eliminated the third cat and added to the explanation. – G-Man Oct 10 '15 at 17:10

If you're more interested in utility rather than being cool or clever, then perl is an easier choice than awk. By and large it will be on every *nix with consistent behaviour, and is easy and free to install on windows. I think it's also less cryptic than awk, and there will be some stats modules you could use if you wanted a halfway house between writing it yourself and something like R. My fairly untested (in fact I know it has bugs but it works for my purposes) perl script took about a minute to write, and I'd guess the only cryptic part would be the while(<>), which is the very useful shorthand, meaning take the file(s) passed as command line arguments, read a line at a time and put that line in the special variable $_. So you could put this in a file called count.pl and run it as perl count.pl myfile. Apart from that it should be painfully obvious what's going on.

$max = 0;
while (<>) {
 $sum = $sum + $_;
 $max = $_ if ($_ > $max);
 $count++;
}
$avg=$sum/$count;
print "$count numbers total=$sum max=$max mean=$avg\n";
share|improve this answer
3  
You haven't shown the median – Peter.O Mar 28 '12 at 14:31

The below sort/awk tandem does it:

sort -n | awk '{a[i++]=$0;s+=$0}END{print a[0],a[i-1],(a[int(i/2)]+a[int((i-1)/2)])/2,s/i}'

(it calculates median as mean of the two central values if value count is even)

share|improve this answer

cat/python only solution - not empty-input proof!

cat data |  python3 -c "import fileinput as FI,statistics as STAT; i = [int(l) for l in FI.input()]; print('min:', min(i), ' max: ', max(i), ' avg: ', STAT.mean(i), ' median: ', STAT.median(i))"
share|improve this answer
    
You haven't shown the median – Peter.O Sep 9 '15 at 22:59
    
@Peter.O fixed. – ravwojdyla Sep 10 '15 at 0:33
    
The statistics module requires python version >= 3.4 – Peter.O Sep 10 '15 at 13:05
    
@Peter.O you are correct - is that a problem? – ravwojdyla Sep 10 '15 at 16:17
    
Its not a problem unless you don't have the appropriate python version. It just make it less portable. – Peter.O Sep 10 '15 at 22:54

The num is a tiny awk wrapper which exactly does this and more, e.g.

$ echo "1 2 3 4 5 6 7 8 9" | num max
9
$ echo "1 2 3 4 5 6 7 8 9" | num min max median mean
..and so on

it saves you from reinventing the wheel in the ultra-portable awk. The docs are given above, and the direct link here (check also the GitHub page).

share|improve this answer
    
Links to obscured web code to be executed in the user computer seems to me like a bad idea. The site that contains the code resides here – BinaryZebra Feb 12 at 6:29
    
Where was this "battletested" code hosted before being put on github all of 4 months ago? I find it extremely suspicious that the link to github has to be peeled from the curl download command. It is much more easy to find out how to financially donate to the developer. Looks like the author of that code is afraid people might go to github and look at the (almost non-existent) history and statistics. Is there any reason to call this battle tested at all, apart from trying to raise money? – Anthon Feb 12 at 6:58
    
@BinaryZeba: updated – coderofsalvation Mar 22 at 6:43
    
@Anthon ok, removed the 'battletested' part. I don't think this is the place for conspiracy FUD. – coderofsalvation Mar 22 at 6:45

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.