Good morning,

I have many blocks of data, containing anywhere from 1 to 8 variables (denoted "CONDx" below) based on user input. I have written a script using awk and grep to extract data to be presented in a column format. This data I have extracted from a larger file, so maybe I need to go a step back for my solution. Anyways, the data looks like this:

> cat file
REF    Data1
COND1  Value1
COND2  Value2
REF    Data2
COND3  Value3
REF    Data3
COND1  Value4
COND3  Value5

My script presents the results in the following column format, which I need to manually modify in the vertical direction to have it line up correctly:

        COND1   COND2   COND3   COND4   COND5   COND6   COND7   COND8
Data1   Value1  Value2  Value3  x       x       x       x       x               
Data2   Value4          Value5  

My question is, is it possible using awk (or sed, whatever) to check if each CONDx is contained in each REF block, if it is print the corresponding "ValueX", and if it is not, print an "x" (or even better a blank) as a placeholder? So the desired output would be:

        COND1   COND2   COND3   COND4   COND5   COND6   COND7   COND8
Data1   Value1  Value2  x       x       x       x       x       x       
Data2   x       x       Value3  x       x       x       x       x
Data3   Value3  x       Value5  x       x       x       x       x

For COND1 for example, part of the script contained:

 grep COND1 file | awk '{print $2} END { if (!NR) print "x" }' > temp.cond1

temp.cond1 to be pasted in the resultant file, but this only prints an "x" in the first row as seen in my output, I understand why it doesn't work, but can't think of new way. I think maybe it's possible to do with an IF statement perhaps? Would appreciate any help.

Thanks for your time.

1 Answer 1


Here is an implementation in awk. It has been some time since I used the language for more than a couple of line program, and thought it would be an interesting exercise.

To run awk with a program you need to specify the -f flag, eg:

awk -f my_program.awk my_data.txt

This implementation only outputs the CONDx variables that it finds in the file.

# Initialize a couple of variables
    fill_value = "xx"
    record_number = 0

# for any line that begins and ends with `foo` save the record
# and then move on to process the next line
/^foo$/ { save_record(); next }

# for any other line, grab the key and data, and mark that the record is valid
    fields[$1] = $1
    record[$1] = $2;
    record[1] = "exists"

# after reading in all of the records, output them
    # sort the fields into alpha order
    delete fields["REF"]

    printf("%-8s", "REF")
    for (field in fields) {
        printf("%-8s", fields[field])
    print ""

    # print the records
    for (i=0; i < record_number; i++) {
        record_name = record_number_str(i, "REF");
        printf("%-8s", records[record_name])

        for (field in fields) {
            record_name = record_number_str(i, fields[field])
            to_print = fill_value
            if (record_name in records)
                to_print = records[record_name]
            printf("%-8s", to_print)
        print ""

function save_record() {
    if (1 in record) {
        delete record[1]
        for (rec in record)
            records[record_number_str(record_number, rec)] = record[rec]
    delete record

# awk only has single dimensional associative arrays.  So we need
# to construct a key for the array that has two dimensions
function record_number_str(record_number, rec) {
    return sprintf("%06d %s", record_number, rec)

I reckon that awk is not the ideal language for this. Better would likely be something like: perl, ruby or python. As a contrast here is python implementation. Note that it is only about 1/2 as many lines:

import fileinput

record = {}
records = []
fields = set()
for line in [l.strip() for l in fileinput.input()]:
    if line == 'foo':
        if record:
            record = {}
        key, value = line.split()
        record[key] = value

# print the header
print("%-8s" % "REF", end="")
for field in sorted(fields):
    print("%-8s" % field, end="")

# print the records
for record in records:
    print("%-8s" % record["REF"], end="")
    for field in sorted(fields):
            print("%-8s" % record.get(field, ''), end="")
  • Appreciate the response and help, I will have to implement this into the script and give it a whirl. Thanks very much for your help.
    – Henry
    Commented Jan 11, 2017 at 15:17
  • Thank you, I have played around with the awk function (haven't been able to get python function to run - I may not have python), and have your awk function working after a little modification on my input. I did have to comment out the asort(fields) line however, I don't see any issues with that. One problem I am having however is that it does not pick up the very last COND / VALUE entry. Basically my input (file) ends with: CONDX ValueX, and then a blank line (no foo afterwards). Any thoughts on how to pick up the last CONDX ValueX entry? Thanks in advance.
    – Henry
    Commented Jan 11, 2017 at 19:10
  • 1
    Call save_record() at the beginning of the END block to commit the remaining record. Also, the python code was there only as an illustration of another possible language for these sorts of problems. Cheers. Commented Jan 11, 2017 at 20:00
  • I just realized, I could echo an additional ' REF' at the end of the input, and that fixed it for me. Appreciate your help immensely, thank you sir.
    – Henry
    Commented Jan 11, 2017 at 20:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .