Add every nth line together

Question

I'm looking at averaging a number of data points given from multiple runs of a command. In this particular case, I'm using the sysbench tool suite.

This example input is from a single sysbench cpu <parameters> run; keep in mind this will be run multiple times as I'm looking to average certain results:

CPU speed:
    events per second:   827.81

General statistics:
    total time:                          30.0021s
    total number of events:              44977

Latency (ms):
         min:                                    2.01
         avg:                                    4.93
         max:                                  733.85
         95th percentile:                       12.98
         sum:                               228990.08

In my example script

(for ((n=0;n<3;n++)); do
    <command> | grep -i <data point lines>;
done) | awk '{ print $(NF) }'

I grab the relevant data points from each run and print (as a placeholder for whatever code I will use) the last field of each line below. This contains numbers I need to work with. What I need is something that will add every nth $(NF) together and then average them, which I can already conceptualize.

Let's say from that example I want to take these points (via grep currently but not fussed on implementation)

events per second:   827.81
avg:                                    4.93
max:                                  733.85
95th percentile:                       12.98

And take the last column from them (via awk '{ print $(NF) }' currently)

827.81
4.93
733.85
12.98

Assuming 4 lines of output per cycle and n cycles I want to add all lines matching the same field (so all avg:'s are summed and averaged, same for max: and other stats.)

add lines 1/5/9/4n
          2/6/10/4n
          3/7/11/4n
          4/8/12/4n
together

I'm trying to use awk (as this code will be used over multiple unix systems and awk is a common factor) but if I'm overcomplicating this and there is a better alternative I'm happy to hear it.

Answer 1

You can get something close to what you want with:

$ awk -F'[: ]' -v OFS=: '{sub("^ *","")};
    /^(events|avg|min|95th)/ { sum[$1] += $NF ; count[$1]++ };
    END { for (i in sum) print i, sum[i]/count[i] }' input.[123]
95th:11.3133
avg:4.53
events:691.143
min:2.31

note: input.1, input.2, and input.3 are copies of your sample input with slightly edited values.

also note: sub() is used in the first line of the script to strip any leading spaces. Because $0 is changed, the field split is re-evaluated, so we're guaranteed to have a usable (short) name in $1. We don't really care what it is, only that it uniquely identifies the type of record we're extracting the value from.

Unfortunately, because this uses associative arrays the order of the output is pseudo-random (i.e. you can't count on the elements coming out in any particular order). To get a consistent order, you can use sort on the first output field, and then cut to get only the values.

$ awk -F'[: ]' -v OFS=: '{sub("^ *","")};
    /^(events|avg|min|95th)/ { sum[$1] += $NF ; count[$1]++ };
    END { for (i in sum) print i, sum[i]/count[i] }' input.[123] |
  sort -t: -k1,1 | cut -d: -f2
11.3133
4.53
691.143
2.31

If you have GNU awk, you can use the asorti() function to sort the indices of the associative array, eliminating the need to pipe into sort and cut.

$ awk -F'[: ]' -v OFS=: '{sub("^ *","")};
    /^(events|avg|min|95th)/ {sum[$1] += $NF ; count[$1]++ };
    END {
      num=asorti(sum,idx);
      for (i=1;i<=num;i++) print sum[idx[i]]/count[idx[i]]
    }' input.[123]
11.3133
4.53
691.143
2.31

Answer 2

write your script as below :

{ for ((n=0; n<4; n++)); do <command>; done; } |\
awk -v t=$n 'i==t{ i=0 }
    /(events per second|avg|max|95th percentile):/{ seen[++i]+=$NF/t }
END{ for(x in seen) print x, seen[x] }'