The Unix package datamash supports the application of several summarizing operations to groups of input lines. For example1, here datamash is used to compute the sums of column 2 for each value in column 1:

$ cat example.csv
$ datamash -t, -g 1 sum 2 < example.csv

Although datamash supports a wide range of functions besides sum (including mean, stddev, median, iqr, min, max, etc.), it is not extensible, AFAICT. IOW, datamash does not support any mechanism for the user to supply his/her own summarizing function.

My question here boils down to: how can this group-wise application of commands be implemented generically on zsh2?

Below is an attempt to specify the question more precisely. (Hopefully this attempt at precision won't render the question incomprehensible.)

First, suppose that foo stands for a (possibly composite) command that emits to stdout lines with the following structure:

i separator payloadij

...where i, the "group index", is some integer, separator is some constant separator sequence (e.g. ,, or $'\t'), and payloadij is some arbitrary text (including the terminating newline). Moreover, assume that the group index i ranges from 1 to N, and that the lines in this output are sorted according to the group index.

For every integer 1 ≤ k ≤ N, let the "k-th group" refer to the content consisting of the segments payloadkj of all the lines (in foo's output) where the group index is k.

Next, suppose that bar stands for a (possibly composite) command that reads lines from stdin and emits a single line to stdout.

Now, let resultk denote the output of applying bar to the k-th group, and let X<bar> stand for some shell construct that invokes bar.

I'm basically looking for a construct X<bar> such that the pipeline

foo | X<bar>

emits to stdout lines of the form

i separator resulti


Supposing that separator is just ,, then the following seems to do what I want

TMPFILE=$( mktemp )
foo | (cat; echo) | while IFS= read -r LINE
    if [[ $GROUPID != $LASTGROUPID ]]
        if [[ -n $LASTGROUPID ]]
            echo -n "$LASTGROUPID$SEPARATOR"
            cat $TMPFILE | bar
        : > $TMPFILE
    echo $PAYLOAD >> $TMPFILE

Basically, this use $TMPFILE to collect the lines in the next group. (I'd prefer to avoid the temporary file, but I don't know how to do it.)

Now I need to figure out a way to implement this as a function that can take the expression denoted by bar as an argument, and use it robustly in the construct given above.

1This example is adapted from one given in the datamash man page.

2Although I am primarily interested in zsh, I have a secondary interest bash case as well.

  • awk and gnuplot are not an option? – Andrew Miloradovsky Mar 18 '16 at 22:06
  • @AndrewMiloradovsky: I've edited my post to give a better idea of the sort of thing I'm looking for. – kjo Mar 18 '16 at 22:51

Doesn't sound to me like a job for a shell. I'd do it in perl/python/ruby... though here awk may be enough:

$ cat sum
paste -sd + - | bc
$ sort -t , -k 1,1 input | awk -F, -v cmd=./sum '
   function out() {printf "%s,", l;close(cmd)}
   NR>1 && $1 != l {out()}
   {print $2 | cmd; l=$1}
   END {if (NR) out()}'
| improve this answer | |

If I've got the idea of what you're looking for: a script analogous to those producing the distribution from a set of samples, but with more accumulation options. I wrote an awk script for that.


It is not exactly what you want but the overlap must be significant. First - indices may be not only integer, second - the only accumulation method is summation. But since it is just a script, you can modify it as you like more easily than say C program.

Finally, such scripts are only applicable if the data set is small enough, for larger sets it is too slow! So a more specialized package is preferable (R etc.).

P.S. To add other accumulators, replace += with a custom function (a la "monad").

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.