Using 'awk' to print a placeholder in several blocks of data

Question

Good morning,

I have many blocks of data, containing anywhere from 1 to 8 variables (denoted "CONDx" below) based on user input. I have written a script using awk and grep to extract data to be presented in a column format. This data I have extracted from a larger file, so maybe I need to go a step back for my solution. Anyways, the data looks like this:

> cat file
foo
REF    Data1
COND1  Value1
COND2  Value2
foo
REF    Data2
COND3  Value3
foo
REF    Data3
COND1  Value4
COND3  Value5
foo

My script presents the results in the following column format, which I need to manually modify in the vertical direction to have it line up correctly:

        COND1   COND2   COND3   COND4   COND5   COND6   COND7   COND8
Data1   Value1  Value2  Value3  x       x       x       x       x               
Data2   Value4          Value5  
Data3

My question is, is it possible using awk (or sed, whatever) to check if each CONDx is contained in each REF block, if it is print the corresponding "ValueX", and if it is not, print an "x" (or even better a blank) as a placeholder? So the desired output would be:

        COND1   COND2   COND3   COND4   COND5   COND6   COND7   COND8
Data1   Value1  Value2  x       x       x       x       x       x       
Data2   x       x       Value3  x       x       x       x       x
Data3   Value3  x       Value5  x       x       x       x       x

For COND1 for example, part of the script contained:

 grep COND1 file | awk '{print $2} END { if (!NR) print "x" }' > temp.cond1

temp.cond1 to be pasted in the resultant file, but this only prints an "x" in the first row as seen in my output, I understand why it doesn't work, but can't think of new way. I think maybe it's possible to do with an IF statement perhaps? Would appreciate any help.

Thanks for your time.

Answer 1

Here is an implementation in awk. It has been some time since I used the language for more than a couple of line program, and thought it would be an interesting exercise.

To run awk with a program you need to specify the -f flag, eg:

awk -f my_program.awk my_data.txt

This implementation only outputs the CONDx variables that it finds in the file.

# Initialize a couple of variables
BEGIN {
    fill_value = "xx"
    record_number = 0
}

# for any line that begins and ends with `foo` save the record
# and then move on to process the next line
/^foo$/ { save_record(); next }

# for any other line, grab the key and data, and mark that the record is valid
{
    fields[$1] = $1
    record[$1] = $2;
    record[1] = "exists"
}

# after reading in all of the records, output them
END {
    # sort the fields into alpha order
    asort(fields)
    delete fields["REF"]

    printf("%-8s", "REF")
    for (field in fields) {
        printf("%-8s", fields[field])
    }
    print ""

    # print the records
    for (i=0; i < record_number; i++) {
        record_name = record_number_str(i, "REF");
        printf("%-8s", records[record_name])

        for (field in fields) {
            record_name = record_number_str(i, fields[field])
            to_print = fill_value
            if (record_name in records)
                to_print = records[record_name]
            printf("%-8s", to_print)
        }
        print ""
    }
}

function save_record() {
    if (1 in record) {
        delete record[1]
        for (rec in record)
            records[record_number_str(record_number, rec)] = record[rec]
        record_number++
    }
    delete record
}

# awk only has single dimensional associative arrays.  So we need
# to construct a key for the array that has two dimensions
function record_number_str(record_number, rec) {
    return sprintf("%06d %s", record_number, rec)
}

I reckon that awk is not the ideal language for this. Better would likely be something like: perl, ruby or python. As a contrast here is python implementation. Note that it is only about 1/2 as many lines:

import fileinput

record = {}
records = []
fields = set()
for line in [l.strip() for l in fileinput.input()]:
    if line == 'foo':
        if record:
            records.append(record)
            record = {}
    else:
        key, value = line.split()
        record[key] = value
        fields.add(key)

# print the header
print("%-8s" % "REF", end="")
fields.remove("REF")
for field in sorted(fields):
    print("%-8s" % field, end="")
print()

# print the records
for record in records:
    print("%-8s" % record["REF"], end="")
    for field in sorted(fields):
            print("%-8s" % record.get(field, ''), end="")
    print()