shell script to do some text manipulation of text file data structure and slight content changes

Question

Apologies in advance for the wall of text, not sure how else to represent the existing data structure.

I have been handed about a years worth of logs collected every hour from a server.

Sadly, fixing the data collection script so it builds files the way they want them going forward was easy, but I still have to convert the thousands of existing files. I am running into some stops here, and suspect that either its beyond my skills, or I am mentally making this way harder than it has to be.

This is the base iteration of one of the runs (it runs every 5 minutes every hour)

2350
 id pool                 type rid rset                  min  max size used load
  5 SUNWtmp_serverxd1z1      pset   1 SUNWtmp_serverxd1z1       104  104  104 0.00 6.25
  4 SUNWtmp_serverxd1z2      pset   2 SUNWtmp_serverxd1z2        16   16   16 0.00 0.91
  0 pool_default         pset  -1 pset_default           24  66K   24 0.00 1.74

 id pool                 type rid rset                  min  max size used load
  5 SUNWtmp_serverxd1z1      pset   1 SUNWtmp_serverxd1z1       104  104  104 5.01 6.21
  4 SUNWtmp_serverxd1z2      pset   2 SUNWtmp_serverxd1z2        16   16   16 0.97 0.91
  0 pool_default         pset  -1 pset_default           24  66K   24 3.73 1.78

output truncated, but it goes on for 50 lines from the prior timestamp, until the next one.

I'm not sure how to display numbers within a block quote, but each run is 50 lines long (they all get combined into a file that is around 14400 lines for each day, with the field in the front of each line being the date derived from the file name.

Here is what they want it to look like. Field position in terms of white space doesn't seem to matter, just relative field position, including the new field "int" which is shown iterating to 2, but would actually only iterate once every 50 lines (the complete data collection run), and then start back at 01.

date     hhmm int id pool                type rid rset                  min  max size used load
20121105 2350 01  5 SUNWtmp_serverxd1z1      pset   1 SUNWtmp_serverxd1z1       104  104  104 0.00 6.25
20121105 2350 01  4 SUNWtmp_serverxd1z2      pset   2 SUNWtmp_serverxd1z2        16   16   16 0.00 0.91
20121105 2350 01  0 pool_default         pset  -1 pset_default           24  66K   24 0.00 1.74

date     hhmm int id pool                type rid rset                  min  max size used load
20121105 2350 02  5 SUNWtmp_serverxd1z1      pset   1 SUNWtmp_serverxd1z1       104  104  104 5.01 6.21
20121105 2350 02  4 SUNWtmp_serverxd1z2      pset   2 SUNWtmp_serverxd1z2        16   16   16 0.97 0.91
20121105 2350 02  0 pool_default         pset  -1 pset_default           24  66K   24 3.73 1.78

I've tried a few sed and awk one liners, but come to the sad realization that I've never had to manipulate text in any way that was more complex than a 1 liner could handle, and at this point I see this file needing something more complex than that.

example of something I tried to play with format:

gawk -vdate=$DATE -vtime=$TIME '{print date " " time $0 }' ./poolstat_original_format.txt

using date and time derived from file name into those two variables.

My prior shell scripting experience is all related to system automation and troubleshooting, never had any real experience with this much moving around of text, so if this is actually an easy problem and I am just over thinking it, great, any useful comments appreciated.

Additional info I meant to add but got pulled away.

The date is derived from the file name coming in. 20121003-poolstat_serverxd1z0.txt The time is the 4 digit numeric every 50 lines.

The int field needs to iterate each time the poolstat is run. Se below for details.

In summary, the only fields that need to be changed:

field 1, the 8 digit date, derived from filename IE: 20121003-poolstat_serverxd1z0.txt field 2 the 4 digit time that is inside the file every 50th line. field 3 the iteration count, as follows: Based on digits 3 and 4 of the 4 digit time. 00-05-10-15-20-25-30-35-40-45-50-55 minute of run.

01-02-03-04-05-06-07-08-09-10-11-12 iteration.

The rest is just printing out existing fields, its getting those onto a line, then awk ( or other) command to print out the other 10 fields, all while keeping track of the current iteration.

And just to keep things complex, the fields in the header line also need 3 new fields:

date hhmm int

the rest of the fields are headers supplied by poolstat.

Answer 1

With a reservation concerning my comment question this should work:

awk -v date=20121105 'NF == 0 {print; next;};
  NF==15 && $2 == "id" {readvar=$1; for (i=1;i<15;i++) $i = $(i+1); NF=14; };
  NF==14 { if (block<2) block++; 
    concatvars=$1; for (i=2;i<11;i++) concatvars=concatvars " " $i;
    print "date hhmm int",concatvars,date,readvar,sprintf("%02d", block),"05"; };
  NF==10 {readvar2=$7;
    print $1,"pset 1",$1,$2,$3,$4,$5,$6,date,readvar,sprintf("%02d", block);};
  NF==15 { 
    print readvar2,$1,"pset 2",$1,$2,$3,$4,$5,$6,date,sprintf("%02d", block),$7;
    nextline=$8; for (i=9;i<15;i++) nextline=nextline " " $i; };
  NF==1 {print nextline,$1}' inputfile

Answer 2

It looks like you have to grab the time from the start of the block and then just add some fields to each line thereafter.

I'm not sure how the iteration should be done, based on your description

awk -v date=20121105 '
    NF == 1 {time = $1; intv = intv%50 + 1; next}
    $1 == "id" {print "date", "hhmm", "int", $0; next}
    NF > 0 {$1 = date OFS time OFS sprintf("%02d", intv) OFS $1}
    1
' file

If you want to make the output prettier, pipe the awk command into column -t