Concatenating some strings in some specific lines of multiple text files into a single file with Bash

Question

I have ~100 text files named *.txt in the following form:

=========================================================
 rho_center   e_center    Mass      Mass_0      R_e
 1.0000e-03 1.1236e-03 1.5379e+00 1.6522e+00 1.0734e+01
     J         T/W       Omega   Omega_Kepler axes_ratio
 0.0000e+00 0.0000e+00 0.0000e+00 3.5265e-02 1.0000e+00
    J/M^2
 0.0000e+00
=========================================================
yy| rhoc  =   1.00000000e-03 ;
yy| rp_re =   1.00000000e+00 ;
yy| A_diff=   1.00000000e+00 ;
yy| Re    =   1.07336268e+01 ;
yy| M0    =   1.65219709e+00 ;
yy| M     =   1.53786062e+00 ;
yy| W     =   2.10814073e-01 ;
yy| T     =   0.00000000e+00 ;
yy| J     =   0.00000000e+00 ;
yy| beta  =   0.00000000e+00 ;
yy| M_R   =   1.43275023e-01 ;
  | === periods in msecs ==========
yy| Pa    =              inf ;
yy| Pe    =              inf ;
yy| td    =   8.46677695e-01 ;
yy| PK    =   8.77582921e-01 ;
yy| MDIV = 301 ; SDIV = 601;

I want to concatenate the values of rhoc and M in a single text file, like this:

rhoc M
0.001 1.537
...    ...
0.004 2.328

I am a layman of bash and I try to use awk to do it but failed. How to write the bash script to do this job?

Answer 1

With gawk, assuming there's only one rhoc/M per file:

gawk '
  BEGIN {print "rhoc M"}
  BEGINFILE {delete field}
  $3 == "=" {field[$2] = $4}
  ENDFILE {
    if (("rhoc" in field) && ("M" in field))
      print field["rhoc"], field["M"]
  }' ./*.txt

That will process the files in the lexicographical order of they names. If using zsh, replace ./*.txt with ./*.txt(n) so that the order be numeric (so file2.txt comes before file10.txt for instance).

Answer 2

For non-gawk use (tested on linux (mawk) and FreeBSD), try

LC_ALL=C awk -vHD="rhoc M" '
BEGIN           {for (MX=m=split(HD, IX); m; m--) FLDS[IX[m]]
                 print HD
                }
FNR == 1        {CNT++
                }               
$2 in FLDS      {OUT[$2, CNT] = $4+0
                }
END             {for (i=1; i<=CNT; i++) {for(j=1; j<=MX; j++) printf "%.6g%s", OUT[IX[j],i], (j==MX?ORS:OFS)}
                }
'  *.txt
rhoc M
0.001 1.53786

It will allow you to extend your data extraction by simply extending the HD variable upfront. It counts up the input files, collects the desired data per file into the OUT array, and in the END section prints all out in the order of the files encountered.

Answer 3

sed :

sed -n '/rhoc/{s/^yy| rhoc *= *\([0-9.e+-]*\) *;/\1/;h;};/ M /{s/^yy| M *= *\([0-9.e+-]*\) *;/\1/;x;G;s/\n/ /p}' sample

in a multiline fashion :

sed -n '
    /rhoc/{
        s/^yy| rhoc *= *\([0-9.e+-]*\) *;/\1/
        h
    }
    / M /{
        s/^yy| M *= *\([0-9.e+-]*\) *;/\1/
        x;G;s/\n/ /p
    }
' sample