how can i pull data based on certain flags for rows of data and create a new file with them

Question

I'm working with Upper Air Balloon data with a format of ....

**254     12      1      JAN    2002
  1  94703  72501  40.87N 72.87W    20   1116**
  2    100    100    250     40  32767      3
  **3           OKX                32767     ms**
  **9   1014     20   -105   -130    310      5**
  5   1008     66    -71    -91  32767  32767
  **4   1000    131    -45    -95    315     20**
  5    946    565    -63   -143  32767  32767
  **4    925    741    -75   -155     20     36**
  5    865   1257   -117   -177  32767  32767
  **4    850   1393   -117   -197    290     56**
  5    839   1493   -107   -187  32767  32767
  5    757   2276   -151   -221  32767  32767
  5    745   2396   -143   -303  32767  32767
  5    727   2581   -145   -285  32767  32767
  4    700   2869   -161   -301    275    190
  5    693   2944   -165   -325  32767  32767
  5    676   3130   -175   -285  32767  32767
  5    668   3219   -179   -359  32767  32767
  5    651   3411   -191   -271  32767  32767
  5    595   4076   -219   -279  32767  32767
  5    574   4338   -225   -305  32767  32767
  5    557   4558   -229   -429  32767  32767
  4    500   5350   -285   -455    260    324
  5    459   5955   -333   -483  32767  32767
  5    438   6281   -359   -459  32767  32767
  5    430   6408   -367   -447  32767  32767
  5    420   6571   -365   -485  32767  32767
  4    400   6910   -387   -537    255    401
  5    370   7439   -429   -559  32767  32767
  5    361   7605   -427   -567  32767  32767
  4    300   8830   -521   -631    250    483
  5    279   9295   -553   -653  32767  32767
  4    250   9990   -569   -679    250    550
  5    236  10354   -569   -689  32767  32767
  4    200  11410   -539   -729    255    545
  5    189  11772   -537   -737  32767  32767
  4    150  13250   -559   -789    260    581
  5    139  13731   -581   -811  32767  32767
  4    100  15790   -623   -843    255    380
254      0      2      JAN    2002
  1  94703  72501  40.87N 72.87W    20   2305
  2    100    179    263     26  32767      3
  3           OKX                32767     ms
  9   1016     20    -13   -113    320     36
  4   1000    150    -13   -123    320     72
  4    925    764    -71   -141    320    144
  5    873   1210   -113   -152  32767  32767
  4    850   1416   -125   -185    340    164
  5    836   1542   -129   -239  32767  32767
  5    787   2002   -119   -289  32767  32767
  4    700   2892   -173   -353    310    164
  5    577   4313   -253   -423  32767  32767
  5    557   4569   -235   -435  32767  32767
  5    536   4848   -257   -377  32767  32767
  4    500   5350   -291   -401    285    344
  5    442   6217   -357   -507  32767  32767
  4    400   6910   -389   -569    275    416
  4    300   8850   -477   -677    270    607
  4    250  10040   -521   -711    265    602
  5    219  10891   -539   -739  32767  32767
  5    215  11010   -511   -731  32767  32767
  4    200  11480   -533   -753    260    581
  4    150  13320   -563   -803  32767  32767
  5    109  15314   -623   -843  32767  32767
  4    100  15850   -621   -841    265    669

Column 1 is the line type indicator (LTI) (LTI) Description 254 is the Header with 0(time 0 or 12Z) 2(day) Jan(month) 2002(year) Indicates a new sounding in the output file 1 give other important info like site lat lon and site elevation 3 Site name and wind measuremen (m/s) 9 Surface Information 4 Mandatory Level 5 Significant Level

I have a text file with all of observed data from 2002 to 2019. Each sounding in the file is the same format. The data im trying to extract is (LTI) 254 (entire row) 1 (entire row) 3 (entire row) 9 (entire row) the remaining data will be flagged by columns 1 and 2 in the file which are mandatory observation pressure levels 4 1000 and remaining row 4 925 and remaining row 4 850 and remaining row

As you go down thru the file time increases. I'm looking to output into a new file that looks like this

254     12      1      JAN    2002
  1  94703  72501  40.87N 72.87W    20   1116
  3           OKX                32767     ms
  9   1014     20   -105   -130    310      5
  4   1000    131    -45    -95    315     20
  4    925    741    -75   -155     20     36
  4    850   1393   -117   -197    290     56
254      0      2      JAN    2002
  1  94703  72501  40.87N 72.87W    20   2305
  2    100    179    263     26  32767      3
  9   1016     20    -13   -113    320     36
  4   1000    150    -13   -123    320     72
  4    925    764    -71   -141    320    144
  4    850   1416   -125   -185    340    164

Thank you

Answer 1

Using Raku (formerly known as Perl6)

~$ raku -ne '.trim-leading.put if .words[0].grep(rx/"**"? <(254 | 1 | 3 | 9)> /) | (.words[0].grep("4" | "**4") && .words[1].grep(rx/1000 | 925 | 850 /)) ;' UABalloon.txt
**254     12      1      JAN    2002
1  94703  72501  40.87N 72.87W    20   1116**
**3           OKX                32767     ms**
**9   1014     20   -105   -130    310      5**
**4   1000    131    -45    -95    315     20**
**4    925    741    -75   -155     20     36**
**4    850   1393   -117   -197    290     56**
254      0      2      JAN    2002
1  94703  72501  40.87N 72.87W    20   2305
3           OKX                32767     ms
9   1016     20    -13   -113    320     36
4   1000    150    -13   -123    320     72
4    925    764    -71   -141    320    144
4    850   1416   -125   -185    340    164

The .grep code above relies on Raku's .words function, which splits on whitespace. Code is included to capture lines starting with "**". The .trim-leading method removes leading whitespace from the output.

You can see this solution proceeds in two steps. To trim off initial-and-terminal asterisks, run the code above, save to a tmp_file.txt, and then run the code below:

~$ raku -pe 's:g/ \** (.+?) \** /$0/;' tmp.txt
254     12      1      JAN    2002
1  94703  72501  40.87N 72.87W    20   1116
3           OKX                32767     ms
9   1014     20   -105   -130    310      5
4   1000    131    -45    -95    315     20
4    925    741    -75   -155     20     36
4    850   1393   -117   -197    290     56
254      0      2      JAN    2002
1  94703  72501  40.87N 72.87W    20   2305
3           OKX                32767     ms
9   1016     20    -13   -113    320     36
4   1000    150    -13   -123    320     72
4    925    764    -71   -141    320    144
4    850   1416   -125   -185    340    164

HTH.

https://raku.org

Answer 2

$ awk '$1 ~ /^(254|1|3|9)$/ || ($1 == 4 && $2 ~ /^(1000|925|850)$/)' input.txt  
254     12      1      JAN    2002
  1  94703  72501  40.87N 72.87W    20   1116
  3           OKX                32767     ms
  9   1014     20   -105   -130    310      5
  4   1000    131    -45    -95    315     20
  4    925    741    -75   -155     20     36
  4    850   1393   -117   -197    290     56
254      0      2      JAN    2002
  1  94703  72501  40.87N 72.87W    20   2305
  3           OKX                32767     ms
  9   1016     20    -13   -113    320     36
  4   1000    150    -13   -123    320     72
  4    925    764    -71   -141    320    144
  4    850   1416   -125   -185    340    164

This prints the input lines where either:

• $1 is one of the allowed values (254, 1, 3, or 9)
• $1 is 4 and $2 is either 1000, 925, or 850.

All other lines are ignored.

NOTE: This does not exactly match your sample output because a) your sample output does not match the description of what you want(*) and b) your comment said you don't want lines beginning with 2.

(*) At least, as best as I can understand it. Your description is very unclear and difficult to parse...it took me several minutes to decipher it, and I'm still not sure it's correct. Either what you want is far more complicated than it seems, or you've written it in a bizarrely complicated fashion.