Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Apologies in advance for the wall of text, not sure how else to represent the existing data structure.

I have been handed about a years worth of logs collected every hour from a server.

Sadly, fixing the data collection script so it builds files the way they want them going forward was easy, but I still have to convert the thousands of existing files. I am running into some stops here, and suspect that either its beyond my skills, or I am mentally making this way harder than it has to be.

This is the base iteration of one of the runs (it runs every 5 minutes every hour)

2350
 id pool                 type rid rset                  min  max size used load
  5 SUNWtmp_serverxd1z1      pset   1 SUNWtmp_serverxd1z1       104  104  104 0.00 6.25
  4 SUNWtmp_serverxd1z2      pset   2 SUNWtmp_serverxd1z2        16   16   16 0.00 0.91
  0 pool_default         pset  -1 pset_default           24  66K   24 0.00 1.74

 id pool                 type rid rset                  min  max size used load
  5 SUNWtmp_serverxd1z1      pset   1 SUNWtmp_serverxd1z1       104  104  104 5.01 6.21
  4 SUNWtmp_serverxd1z2      pset   2 SUNWtmp_serverxd1z2        16   16   16 0.97 0.91
  0 pool_default         pset  -1 pset_default           24  66K   24 3.73 1.78

output truncated, but it goes on for 50 lines from the prior timestamp, until the next one.

I'm not sure how to display numbers within a block quote, but each run is 50 lines long (they all get combined into a file that is around 14400 lines for each day, with the field in the front of each line being the date derived from the file name.

Here is what they want it to look like. Field position in terms of white space doesn't seem to matter, just relative field position, including the new field "int" which is shown iterating to 2, but would actually only iterate once every 50 lines (the complete data collection run), and then start back at 01.

date     hhmm int id pool                type rid rset                  min  max size used load
20121105 2350 01  5 SUNWtmp_serverxd1z1      pset   1 SUNWtmp_serverxd1z1       104  104  104 0.00 6.25
20121105 2350 01  4 SUNWtmp_serverxd1z2      pset   2 SUNWtmp_serverxd1z2        16   16   16 0.00 0.91
20121105 2350 01  0 pool_default         pset  -1 pset_default           24  66K   24 0.00 1.74

date     hhmm int id pool                type rid rset                  min  max size used load
20121105 2350 02  5 SUNWtmp_serverxd1z1      pset   1 SUNWtmp_serverxd1z1       104  104  104 5.01 6.21
20121105 2350 02  4 SUNWtmp_serverxd1z2      pset   2 SUNWtmp_serverxd1z2        16   16   16 0.97 0.91
20121105 2350 02  0 pool_default         pset  -1 pset_default           24  66K   24 3.73 1.78

I've tried a few sed and awk one liners, but come to the sad realization that I've never had to manipulate text in any way that was more complex than a 1 liner could handle, and at this point I see this file needing something more complex than that.

example of something I tried to play with format:

gawk -vdate=$DATE -vtime=$TIME '{print date " " time $0 }' ./poolstat_original_format.txt

using date and time derived from file name into those two variables.

My prior shell scripting experience is all related to system automation and troubleshooting, never had any real experience with this much moving around of text, so if this is actually an easy problem and I am just over thinking it, great, any useful comments appreciated.

Additional info I meant to add but got pulled away.

The date is derived from the file name coming in. 20121003-poolstat_serverxd1z0.txt The time is the 4 digit numeric every 50 lines.

The int field needs to iterate each time the poolstat is run. Se below for details.

In summary, the only fields that need to be changed:

field 1, the 8 digit date, derived from filename IE: 20121003-poolstat_serverxd1z0.txt field 2 the 4 digit time that is inside the file every 50th line. field 3 the iteration count, as follows: Based on digits 3 and 4 of the 4 digit time. 00-05-10-15-20-25-30-35-40-45-50-55 minute of run.

01-02-03-04-05-06-07-08-09-10-11-12 iteration.

The rest is just printing out existing fields, its getting those onto a line, then awk ( or other) command to print out the other 10 fields, all while keeping track of the current iteration.

And just to keep things complex, the fields in the header line also need 3 new fields:

date hhmm int

the rest of the fields are headers supplied by poolstat.

share|improve this question
    
Where does the part "20121105 2350 01 5" come from? "2350" is read but the rest? "01" is increased? "5" is always "5"? Or are they time? "20121105" is date? –  Hauke Laging May 6 '13 at 15:39
    
Is there any use in showing more than two or three input blocks? –  Hauke Laging May 6 '13 at 16:14
    
I've updated the question some to add the extra info that should have been there before. –  Logan Stout May 6 '13 at 18:45
    
"Additional info"? Are you aware that I wrote some non-trivial working code and now you have changed both the input and the output? Next time have a look at your posting and correct it before others start working on it... –  Hauke Laging May 6 '13 at 19:16
add comment

2 Answers

up vote 1 down vote accepted

With a reservation concerning my comment question this should work:

awk -v date=20121105 'NF == 0 {print; next;};
  NF==15 && $2 == "id" {readvar=$1; for (i=1;i<15;i++) $i = $(i+1); NF=14; };
  NF==14 { if (block<2) block++; 
    concatvars=$1; for (i=2;i<11;i++) concatvars=concatvars " " $i;
    print "date hhmm int",concatvars,date,readvar,sprintf("%02d", block),"05"; };
  NF==10 {readvar2=$7;
    print $1,"pset 1",$1,$2,$3,$4,$5,$6,date,readvar,sprintf("%02d", block);};
  NF==15 { 
    print readvar2,$1,"pset 2",$1,$2,$3,$4,$5,$6,date,sprintf("%02d", block),$7;
    nextline=$8; for (i=9;i<15;i++) nextline=nextline " " $i; };
  NF==1 {print nextline,$1}' inputfile
share|improve this answer
add comment

It looks like you have to grab the time from the start of the block and then just add some fields to each line thereafter.

I'm not sure how the iteration should be done, based on your description

awk -v date=20121105 '
    NF == 1 {time = $1; intv = intv%50 + 1; next}
    $1 == "id" {print "date", "hhmm", "int", $0; next}
    NF > 0 {$1 = date OFS time OFS sprintf("%02d", intv) OFS $1}
    1
' file

If you want to make the output prettier, pipe the awk command into column -t

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.