AWK Search massive file and write to variable name

Question

I want to pull data that I will need to reformat from a simulation file I am working with to prepare it as input for running other simulations. The target data is an optimized chemical z-matrix. The only difference from the example in the link is that the digits are stored as variables under the z-matrix.

I'm working with files which can span thousands to hundreds of thousands of lines, and have throughout it other intermediate z-matrices which are extremely similar to my target data. However, there is a unique leading line and a line after the data I need so the data in between those two lines should be relatively simple to extract. Here is an example:

 Final structure in terms of initial Z-matrix:
 Cl
 C,1,B1
 C,2,B2,1,A2
 H,2,B3,1,A3,3,D3,0
 H,2,B4,1,A4,3,D4,0
 C,3,B5,2,A5,1,D5,0
 C,6,B6,3,A6,2,D6,0
 C,7,B7,6,A7,3,D7,0
 H,3,B8,2,A8,1,D8,0
 H,3,B9,2,A9,1,D9,0
 H,7,B10,6,A10,3,D10,0
 H,7,B11,6,A11,3,D11,0
 H,8,B12,7,A12,6,D12,0
 H,8,B13,7,A13,6,D13,0
 H,8,B14,7,A14,6,D14,0
 O,6,B15,3,A15,2,D15,0
      Variables:
 B1=1.81746475
 B2=1.52136867
 A2=110.80057513
 B3=1.0898967
 A3=106.92512231
 D3=-121.94499481
 B4=1.08989406
 A4=106.92581701
 D4=121.94497834
 B5=1.52808963
 A5=111.92359259
 D5=179.99770382
 B6=1.523193
 A6=116.49970868
 D6=179.97424974
 B7=1.52739317
 A7=113.56269053
 D7=179.98802896
 B8=1.09816794
 A8=110.50682514
 D8=58.2854688
 B9=1.09816384
 A9=110.50888758
 D9=-58.28349045
 B10=1.10022643
 A10=107.84652382
 D10=56.40290615
 B11=1.10022793
 A11=107.84460667
 D11=-56.42958848
 B12=1.09398015
 A12=110.97242167
 D12=-59.62466169
 B13=1.09473047
 A13=110.53459142
 D13=179.99742235
 B14=1.09397826
 A14=110.9720435
 D14=59.61905862
 B15=1.21736254
 A15=121.22780588
 D15=-0.02140167
 1\1\GINC-C0959\FOpt\RB3LYP\6-31G(d)\C5H9Cl1O1\SKYLERS\10-Sep-2013\0\\#

I also need to reformat this data, replacing the commas in the first part with spaces and the variables need to be left indented letters followed by right indented numbers. I also want to set up this script to write this data to a new file based on the name of the input file.

My goals in summary:

1. open file abc.out
2. start reading after "final structure in terms of initial Z-matrix:"
3. reformat Z-matrix
4. reformat variables
5. write to file abc.cm

So far these are some of the awk portions i was thinking of using:

  {FS=","};{OFS=" "};{print $0}
  {FS="="};{printf "%-4s%13.8f", $1, $2}

What I still haven't figured out:

• How do I write to a file based on the input file name?
• How do I read only between the first and last lines?
• How should I split this at "Variables:"?

Answer 1

Basically, you'll be coding a little state machine:

awk '
    BEGIN { 
        FS = ","
        OFS = " "    # this is the default
    }

    # create the output file name
    # on the first line of the input, the FILENAME variable will be populated
    FNR == 1 {
        f = FILENAME
        sub(/\.out/,".cm",f)
    }

    # I assume this is the magic closing line.
    # All the backslashes and regular-expression metacharacters 
    # have to be backslash-escaped
    /1\\1\\GINC-C0959\\FOpt\\RB3LYP\\6-31G\(d\)\\C5H9Cl1O1\\SKYLERS\\10-Sep-2013\\0\\\\#/ {
        print "got end"
        exit
    }

    started && /Variables:/ {
        variables = 1
        FS = "="
        next
    }

    started && !variables {
        # do stuff with comma-separated lines
        # rewrite the file using space as separator
        # this looks weird, but it forces awk to re-write the line using OFS
        $1 = $1
        print > f
    }
    started && variables {
        # do stuff with "="-separated lines
        # the FS here is "=", so there should be 2 fields.
        printf "%-5s %15.8f\n", $1, $2 > f
    }

    !started && /Final structure in terms of initial Z-matrix/ {
        started = 1
    }
' abc.out

Given your input, this produces the file "abc.cm"

Cl
C 1 B1
C 2 B2 1 A2
H 2 B3 1 A3 3 D3 0
H 2 B4 1 A4 3 D4 0
C 3 B5 2 A5 1 D5 0
C 6 B6 3 A6 2 D6 0
C 7 B7 6 A7 3 D7 0
H 3 B8 2 A8 1 D8 0
H 3 B9 2 A9 1 D9 0
H 7 B10 6 A10 3 D10 0
H 7 B11 6 A11 3 D11 0
H 8 B12 7 A12 6 D12 0
H 8 B13 7 A13 6 D13 0
H 8 B14 7 A14 6 D14 0
O 6 B15 3 A15 2 D15 0
B1         1.81746475
B2         1.52136867
A2       110.80057513
B3         1.08989670
A3       106.92512231
D3      -121.94499481
B4         1.08989406
A4       106.92581701
D4       121.94497834
B5         1.52808963
A5       111.92359259
D5       179.99770382
B6         1.52319300
A6       116.49970868
D6       179.97424974
B7         1.52739317
A7       113.56269053
D7       179.98802896
B8         1.09816794
A8       110.50682514
D8        58.28546880
B9         1.09816384
A9       110.50888758
D9       -58.28349045
B10        1.10022643
A10      107.84652382
D10       56.40290615
B11        1.10022793
A11      107.84460667
D11      -56.42958848
B12        1.09398015
A12      110.97242167
D12      -59.62466169
B13        1.09473047
A13      110.53459142
D13      179.99742235
B14        1.09397826
A14      110.97204350
D14       59.61905862
B15        1.21736254
A15      121.22780588
D15       -0.02140167

Answer 2

Here is Python Script :

#!/usr/bin/env python
from __future__ import print_function
import sys

StartStr = 'Final structure in terms of initial Z-matrix'
StopStr = '1\\1\\GINC-C0959\\FOpt\\RB3LYP\\6-31G(d)\\C5H9Cl1O1\\SKYLERS\\10-Sep-2013\\0\\\\#'

def main():
        v,start = 0,0
        for line in InputFile:
                line = line.strip()
                if StartStr in line: start = 1; continue
                if StopStr in line: break
                if start:
                    if v:  print('\t'.join(line.split('=')))
                    else:
                        if "Variables" in line: v = 1; print(); continue
                        print(' '.join(line.split(',')))

if __name__ == '__main__':
    if len(sys.argv) != 2:
        print( "\nUsage:\t",sys.argv[0],'<InputFile>\n',file=sys.stderr )
    else:
        try:
                ## create the output file name
                outputFile=sys.argv[1].split('.')[0],".cm"
                o = ''.join(outputFile)
                print("Your Final Output Saved in:- ",o)
                with open(sys.argv[1],'r') as InputFile:
                         sys.stdout = open(o,'w')
                         main()
        except:
                print("Problem with Opening file",sys.argv[1],file=sys.stderr)

Answer 3

This can also be done using Perl's flip-flop operator. Type the following in a shell:

INFILE="abc.out"           #Quotes only necessary ...
OUTFILE="${INFILE%.*}".cm  # ... if you have spaces in the file names
perl -nle '
    if(m{\QFinal structure in terms of initial Z-matrix:\E} ..
       m{\Q1\1\GINC-C0959\FOpt\RB3LYP\6-31G(d)\C5H9Cl1O1\SKYLERS\10-Sep-2013\0\\#\E}){
        (s/,/ /g or !/=|:/) and print;
        /([^=]+)=([^=]+)/ and printf "%-4s %13.8f\n", $1,$2
     }
     ' "$INFILE" > "$OUTFILE"

Answer 4

Given:

• "Final", "Variables" and "#" (at the end of the line) divide the sections
• variables are up-to 5 characters long, values are up-to 15 charactes long

Put your sample into in1.txt, added some dummy lines around (in2.txt)

[root@me]$ ls -l
-rw------- 1 root root 1133 Sep 24 04:10 in1.txt
-rw------- 1 root root 4522 Sep 24 04:12 in2.txt
[root@me]$ find . -name "in*.txt" -type f|xargs -n1 -I{} sed -n '/^ *Final/,/#$/{/Final\|Variables\|#$/d;
s/^ *//;s/,/ /g;s/=/====================/;s/^\(.\{5\}\)=*\(.\{15\}\)/\1\2/;s/=/ /g;w{}.out
}' {}
[root@me]$ ls -l
-rw------- 1 root root 1133 Sep 24 04:10 in1.txt
-rw------- 1 root root 1169 Sep 24 04:20 in1.txt.out
-rw------- 1 root root 4522 Sep 24 04:12 in2.txt
-rw------- 1 root root 1169 Sep 24 04:20 in2.txt.out

Answer 5

This is another awk version.

awk -f- <<\EOF data
    FNR==1 { f = FILENAME".new" }
    /Final structure in terms of initial Z-matrix:/ {
        FS=","
        while ( getline > 0 ) {
            if ( $0 ~ /Variables:/ ) break
            $1=$1
            print $0 > f
        }
        FS="="
        while ( getline > 0 ) {
            if( NF == 2 ) {
                printf "%-5s%15.8f\n", $1, $2 > f
            } else {
                break
            }
        }
    }
EOF