AWK: To divide a column in userdefined chunksize; and count+sum each entry for every chunk; give average per entry for all chunk

Question

I need help to process a column entry via awk. Below are the operations i want to try:

• To divide a column in user-defined chunk size;
• count and sum each entry for every chunk to eventually give average per entry which will be eventually the chunk size.

For instance, below is a list:

1
2
3
4
5
6
7
8
9
10
11
12

Here, I want to use a chunk size of 4 (but in my case it can vary from case to case):

• chunk1

  1
  2
  3
  4
  

• chunk2

  5
  6
  7
  8
  

• chunk3

  9
  10
  11
  12
  

After processing, I would like to have:

5
6
7
8

which is the average for the entries in position 1, 2, 3 and 4, respectively, across all chunks.

Answer 1

The following awk program would do the job. It assumes the data is stored in data.txt in the first column (but can easily be adapted for any other column). It also assumes there are no empty columns, and only complete chunks.

awk -v cs=4 '{if ((i=NR%cs)==0) {n_ch++; i=cs};buf[i]+=$1;} END{for (i=1;i<=cs;i++) printf "%d\n",buf[i]/n_ch}' data.txt

The chunk size is passed to awk via the -v cs=size statement.

It will, for each line, determine the "entry number within the chunk", i, via i = "line number" modulo "chunk size", and sum the entries into an array buf. Whenever one chunk is complete, the chunk counter n_ch is increased.

In the end, we print the average for all entry numbers.

Answer 2

awk -v cs=4 '
  BEGIN {
    "(wc -l <" ARGV[1] ")" | getline nl
    nc = sprintf("%d", nl/cs)
  }
  { a[NR%cs] += $1 }
  NR>nl-cs { print a[NR%cs]/nc } 
' file
5
6
7
8

• in the begin block we determine the lines in the file and store it in nl variable.
• then we get the number of chunks and store it in nc variable.
• accumulate the running sum in the array indexed on the modulus of line number % chunk size.
• then when the line number crosses the threshold of nl-cs, meaning we are entering the last chunk, we begin printinh the results.

Alternatively, if we don't want to precompute the file length and number of chunks we can do this:

awk -v cs=4 '
  { a[NR] = $1 }
  END {
    for (i=1; i<=cs; i++) {
      k = s = 0
      for (j=i; j<=NR; j+=cs) {
        s += a[j]; k++
      }
      print s/k
    }
  }
' file

Answer 3

An alternative awk that just chunks as it goes and counts elements in each chunk so if the record count is not an exact chunk multiple this doesn't matter, it's still the average

awk -v ch=4 '{k=(NR-1)%ch; n[k]++; un[k]+=$1}
  END{for (k in un) print "Line "k+1" has "n[k]" elements totalling "un[k]" and average "un[k]/n[k]}' file

Line 1 has 3 elements totalling 15 and average 5
Line 2 has 3 elements totalling 18 and average 6
Line 3 has 3 elements totalling 21 and average 7
Line 4 has 3 elements totalling 24 and average 8

Although the END needn't be descriptive

  END{for (k in un) print un[k]/n[k]}' file

5
6
7
8

and if you want to guarantee the output sequence

  END{for (k=0; k<ch; k++) print un[k]/n[k]}' file