1

I have a data file that I am preprocessing before sending it to gnuplot. It consists of a header line and multiple rows of data, with many columns of data types. The first three columns are always of the same type and order. The total number of columns is constant within a file, but will not be constant between files. The seqno is not guaranteed to start at 1, but will always be monotonically increasing.

I have a script to insert a header line where I want in the data file, but I want to be able to alter that header based on the match that I am currently in. Specifically, I want to prepend my $i variable to the 4th, 5th, ..endth columns of my header. Apart from this, the header is identical in each placement.

This will run as a script, so if I need to preprocess the header to find out how many columns it has, then that can be easily done.

My current script, without the desired header replacement is (additional header examples at the end):

header=$(head -n1 $input)
awk -v i=3 -v hdr="$header" 'NR>1 && $i!=p {print "\n\n"hdr}{p=$i} 1' ${input} > ${output}

My input example is:

#filename       seqno   phasename       a       b       c       scale   Rwp
blah_001.xye    1       corundum        3       3       12      0.001   3
blah_002.xye    2       corundum        3.1     3.1     12.1    0.002   3.5
blah_003.xye    3       corundum        3.2     3.2     12.2    0.001   3.1
blah_001.xye    2       silcon_NIST     5.4     5.4     5.4     0.002   3
blah_002.xye    3       silcon_NIST     5.41    5.41    5.41    0.004   3.5
blah_003.xye    4       silcon_NIST     5.42    5.42    5.42    0.002   3.1

My current output is:

#filename       seqno   phasename       a       b       c       scale   Rwp
blah_001.xye    1       corundum        3       3       12      0.001   3
blah_002.xye    2       corundum        3.1     3.1     12.1    0.002   3.5
blah_003.xye    3       corundum        3.2     3.2     12.2    0.001   3.1


#filename       seqno   phasename       a       b       c       scale   Rwp
blah_001.xye    2       silcon_NIST     5.4     5.4     5.4     0.002   3
blah_002.xye    3       silcon_NIST     5.41    5.41    5.41    0.004   3.5
blah_003.xye    4       silcon_NIST     5.42    5.42    5.42    0.002   3.1

My desired output is:

#filename       seqno   phasename       corundum_a       corundum_b       corundum_c       corundum_scale   corundum_Rwp
blah_001.xye    1       corundum        3       3       12      0.001   3
blah_002.xye    2       corundum        3.1     3.1     12.1    0.002   3.5
blah_003.xye    3       corundum        3.2     3.2     12.2    0.001   3.1


#filename       seqno   phasename       silcon_NIST_a       silcon_NIST_b       silcon_NIST_c       silcon_NIST_scale   silcon_NIST_Rwp
blah_001.xye    2       silcon_NIST     5.4     5.4     5.4     0.002   3
blah_002.xye    3       silcon_NIST     5.41    5.41    5.41    0.004   3.5
blah_003.xye    4       silcon_NIST     5.42    5.42    5.42    0.002   3.1

What I want to do: How can I alter the awk variable hdr inside awk to prepend the $i variable to the 4th - endth columns of the variable hdr before inserting it to the input file?

Some other example headers in other files for processing

#filename   seqno   phasename   temp    temp_err    csL csL_err csG csG_err strL    strL_err    strG    strG_err    B_Na    B_Na_err    B_Mg    B_Mg_err    B_F B_F_err B_H B_H_err B_O B_O_err B_Fe    B_Fe_err    F_occ   F_occ_err   Na_x    Na_x_err    Na_z    Na_z_err    F1_x    F1_x_err    F1_y    F1_y_err    F1_z    F1_z_err    F2_x    F2_x_err    F2_z    F2_z_err    a1  a1_err  a2  a2_err  a3  a3_err  a4  a4_err  a5  a5_err  a6  a6_err  a7  a7_err  s1  s1_err  s2  s2_err  s3  s3_err  a   a_err   b   b_err   c   c_err   al  al_err  be  be_err  ga  ga_err  volume  volume_err  mass    mass_err    MAC MAC_err density density_err LAC LAC_err Lvol    Lvol_err    e0  e0_err  scale   scale_err   wt% wt%_err num_area    num_area_err    r_bragg r_bragg_err r_wp    r_wp_err    r_exp   r_exp_err   gof gof_err
#filename   seqno   phasename   csL csL_err strG    strG_err    a   a_err   b   b_err   c   c_err   al  al_err  be  be_err  ga  ga_err  volume  volume_err  mass    mass_err    MAC MAC_err density density_err LAC LAC_err Lvol    Lvol_err    e0  e0_err  scale   scale_err   wt% wt%_err num_area    num_area_err    r_bragg r_bragg_err r_wp    r_wp_err    r_exp   r_exp_err   gof gof_err
#filename   seqno   phasename   csG strL    F1_x    F1_y        F1_z    volume      gof 
2
$ awk -f script.awk file
#filename       seqno   phasename       corundum_a      corundum_b      corundum_c      corundum_scale  corundum_Rwp
blah_001.xye    1       corundum        3       3       12      0.001   3
blah_002.xye    2       corundum        3.1     3.1     12.1    0.002   3.5
blah_003.xye    3       corundum        3.2     3.2     12.2    0.001   3.1

#filename       seqno   phasename       silcon_NIST_a   silcon_NIST_b   silcon_NIST_c   silcon_NIST_scale       silcon_NIST_Rwp
blah_001.xye    2       silcon_NIST     5.4     5.4     5.4     0.002   3
blah_002.xye    3       silcon_NIST     5.41    5.41    5.41    0.004   3.5
blah_003.xye    4       silcon_NIST     5.42    5.42    5.42    0.002   3.1

Where script.awk is

BEGIN   { OFS = "\t" }

/^#/    {
    # save header fields

    for (i = 1; i <= NF; ++i)
        header[i] = $i

    next
}

# if column 2 contains a lower number than the previous line
# (or if no previous line with data), then output header
$2 < col2 || !col2 {
    # output blank line if needed
    if (print_blank) {
        print ""
    }
    print_blank = 1

    # print first three headers as-is
    for (i = 1; i <= 3; ++i)
        printf("%s%s", header[i], OFS)

    # prepend column three to remaining headers
    for (i = 4; i < NF; ++i)
        printf("%s_%s%s", $3, header[i], OFS)
    printf("%s_%s%s", $3, header[NF], ORS)
}

# print all lines and save value from column 2
{ col2 = $2; print }

The script saves the header from the input data in the array header. When we find a second column whose value is lower than the previous line's column two value, we output a new header before outputting the data. The header is preceded by a blank line unless it's the first header. The variable column names take their names from the third field.

The script takes no parameters.

  • Thanks. You've manually prepended the thing that I want to the header line. This is the exact process I want to automate, as the header line can have different numbers of columns depending on the data that was outputted. Although I now know the function of the trailing 1. – masher Aug 3 '18 at 8:03
  • @masher Do you have another example with more columns? It would be easy to add more a, b, c, d columns depending on the number of columns in the input data. – Kusalananda Aug 3 '18 at 8:35
  • The number of columns is arbitrary, and the name of each column can also vary, except for the first 3. This is why I want to automate the prepending. Adding some examples in the next couple of minutes – masher Aug 3 '18 at 8:38
  • @masher See updated answer. – Kusalananda Aug 3 '18 at 9:00
  • Ta a lot. I changed two things to make it do exactly what I want: 1) print "\n" in the print_blank section to give two blank lines 2) NR>1 && $i!=p { all of your stuff }{p=$i} to make it work with some of my other seqno values; Some of them were like 000, 001, 002... and yours was splitting on the 000 and 001 – masher Aug 3 '18 at 9:22
1

If the possibility exists that groups of phasenames start with seqno greater than the last one's, it might not work out to rely on seqno, but mayhap better on phasename. You may want to try this adaption of Kusalananda's proposal:

    awk '
    FNR == 1        {split ($0, header)
                     next
                    }

    $3 != LAST      {printf TMPRS; TMPRS = ORS
                     for (i = 1; i <= NF; ++i) printf ("%s%s%s", (i>3)?$3"_":_, header[i], (i==NF?ORS:OFS))
                    }

                    {LAST = $3
                     print
                    }
    ' OFS="\t" filename1 filename2
  • Thanks for you addition. I ended up using a comparison based on phasename: NR>1 && $i!=p { all of the stuff }{p=$i; print} – masher Aug 6 '18 at 14:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.