How can I turn ugly output into pretty, useful information?

Question

How can I turn this ugly output into pretty, useful data?

The output:

/* ---------- TA#box#AbC_p ---------- */

insert_job: TA#box#AbC_p    job_type: a
#owner: bob
permission: gx
date_conditions: 1
days_of_week: su
start_times: "16:15"
run_window: "16:15-17:30"
description: "Job AbC that runs at 4:15PM on Sundays, and should end before 5:30PM"

    /* ---------- TA#cmd#EfGJob_p ---------- */

    insert_job: TA#cmd#EfGJob_p    job_type: b
    box_name: TA#box#AbC_p
    command: /path/to/shell/script.sh
    machine: vm_machine1
    #owner: alex
    permission: gx
    date_conditions: 2
    run_window: "16:20-16:30"
    description: "job EfG that runs within box AbC"
    term_run_time: 60
    std_out: /path/to/log.log
    std_err: /path/to/err.log
    alarm_if_fail: 1
    profile: /path/to/profile

and so on, for a long time. #cmd# jobs are sometimes under a #box#. If they are under a #box#, the #cmd# section is indented.

My ideal output would be something like:

"Job Name", "Time", "Schedule", "Machine", "Description", "Command"
"TA#box#AbC_p", "16:15", "su", "", "Job AbC that runs at 4:15PM on Sundays, and should end before 5:30PM", ""
"TA#cmd#EfGJob_p", "16:15", "su", "vm_machine1", "job EfG that runs within box AbC", "/path/to/shell/script.sh"

I'm trying awk, perl and grep, but I'm having trouble keeping all the info for one "section" together before I print the CSV line.

Answer 1

I'd use Perl for that, or at least awk.

perl -ne '
    BEGIN {
        print "\"Job Name\", \"Time\", \"Schedule\", \"Machine\", \"Description\", \"Command\", \"\n";
    }
    chomp; s/^\s+//; s/\s+$//;
    if (($_ eq "" || eof) && exists $fields{insert_job}) {
        print "\"", join("\", \"", @fields{qw(insert_job start_times days_of_week machine description command)}), "\"\n";
        delete @fields{qw(insert_job)};
    }
    if (/^([^ :]+): *(.*)/) {$fields{$1} = $2}
'

Explanations:

• The BEGIN block is run once at the beginning of the script, the rest runs for every input line.
• The line that begins with chomp strips off leading and trailing whitespace.
• The first if line triggers on empty lines (paragraph separators), if the field insert_job is present.
• The delete line removes the insert_job field. Add other field names that you don't want to spill over from one paragraph to the next.
• The last if line stores fields.

Answer 2

A little terrible sed oneliner:

sed -n  \
# we divide out incoming text to small parts, 
# each one as you mentioned from /---.*box.*/ to /profile/
'/---.*box.*/,/profile/{
     # inside of each part we do following things:
     # if string matches our pattern we extract 
     # the value and give it some identifier (which you
     # can see is "ij", "st" and so on)
     # and we copy that value with identifier to hold buffer,
     # but we don't replace the content of hold buffer
     # we just append (capital H) new var to it
     /insert_job/{s/[^:]*: /ij"/;s/ .*/",/;H};
     /start_times/{s/[^:]*: /st/;s/$/,/;H};
     /days_of_week/{s/[^:]*: /dw"/;s/$/",/;H};
     /machine/{s/[^:]*: /ma"/;s/$/",/;H};
     /description/{s/[^:]*: /de/;s/$/,/;H};
     /command/{s/[^:]*: /co"/;s/$/",/;H};
     # when line matches next pattern (profile)
     # we think that it is the end of our part,
     # therefore we delete the whole line (s/.*//;)
     # and exchange the pattern and hold buffers (x;)
     # so now in pattern buffer we have several strings with all needed variables
     # but all of them are in pattern space, therefore we can remove
     # all newlines symbols (s/\n//g;). so it is just one string 
     # with a list of variables
     # and we just need to move to the order we want,
     # so in this section we do it with several s commands.
     # after that we print the result (p)
     /profile/{s/.*//;x;s/\n//g;s/ij\("[^"]*box[^"]*",\)/\1/;
          s/,\(.*\)st\("[^"]*",\)\(.*ij"[^"]*",\)/,\2\1\3\2/;
          s/\([^,]*,[^,]*,\)\(.*\)dw\("[^"]*",\)\(.*ij"[^"]*",[^,]*,\)/\1\3\2\4\3/;
          s/de/"",/;s/ij/""\n/;
          s/\(\n[^,]*,[^,]*,[^,]*,\)\(.*\)ma\("[^"]*",\)/\1\3\2/;
          s/co\("[^"]*"\),\(.*\)/\2\1/;s/de//;p}
     };
     # the last command just adds table caption and nothing more.
     # note: if you want to add some new commands,
     # add them before this one
     1i"Job Name", "Time", "Schedule", "Machine", "Description", "Command"'

I wrote it as field order may vary in different boxes but profile is always last one. In case the order is always the same it would be a little bit easier.