4

I am producing a huge file which includes sections like this:

~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~ Gradients ~~~~~~~~
~~~~~~~~~ x y z ~~~~~~~~~~
~ ~
~ H         1      0.00781      0.00108      0.00038 ~
~ H         2      0.01271     -0.01507      0.02839 ~
~ C         1     -0.05015     -0.01803      0.01588 ~
~ O         1      0.01733      0.03089     -0.04611 ~
~ O         2      0.01230      0.00114      0.00147 ~
~ ~
~~~~~~~~~~~~~~~~~~~~~~~~

I need to extract these numbers (x y z):

0.00781      0.00108      0.00038
0.01271     -0.01507      0.02839
-0.05015     -0.01803      0.01588
0.01733      0.03089     -0.04611 
0.01230      0.00114      0.00147

I wrote the following script:

awk '/z ~/ {for(i=1; i<=6; i++) {getline; print $4, $5, $6}}' filename

But it gives me a blank line due to the "~ ~" line.

In other words, each time I find the /z ~/ pattern, I want to skip another line (pattern + 1) and only print the content from five other lines (pattern +2 +3 +4 +5 +6). And of course it needs to be a repeating operation (doing it on and on, hundreds of thousands of times).

3
  • Maybe: awk '/z ~/ {n=1}; n && ++n >= 4 && n <= 7' Jun 6, 2017 at 12:20
  • 1
    Would it be fine to just match those lines with numbers? Jun 6, 2017 at 20:18
  • As a commentary/review on design, if you're producing this file you may want to reconsider and use a more standard format or even put the information in an actual database. It's good to know how to use Awk to extract meaningful data from offbeat file formats; it's better not to have to do so. :)
    – Wildcard
    Jun 7, 2017 at 1:56

6 Answers 6

6

awk solution:

awk '/z ~/{ n=NR+2 }n && n<=NR && NR<(n+5){ print $4,$5,$6 }' file | column -t

The output:

0.00781   0.00108   0.00038
0.01271   -0.01507  0.02839
-0.05015  -0.01803  0.01588
0.01733   0.03089   -0.04611
0.01230   0.00114   0.00147

  • NR - current record number

  • n=NR+2 - n here points to the "starting" line number after the pattern line

2
  • You really, really, REALLY hate whitespace, don't you. :) Jun 6, 2017 at 12:28
  • @SatoKatsura, nope. I have worked with older input version. Updated Jun 6, 2017 at 12:31
4

The simplest solution would be to just add another getline and then get 5 lines instead of 6:

$ awk '/z ~/ {getline;for(i=1; i<=5; i++) {getline; print $4, $5, $6}}' file
0.00781 0.00108 0.00038
0.01271 -0.01507 0.02839
-0.05015 -0.01803 0.01588
0.01733 0.03089 -0.04611
0.01230 0.00114 0.00147

Personally, I would have done it in a slightly different way though:

$ awk '/z ~/{f=2;} /~ ~/{f--}; (f==1 && NF>5){print $4, $5, $6} ' file
0.00781 0.00108 0.00038
0.01271 -0.01507 0.02839
-0.05015 -0.01803 0.01588
0.01733 0.03089 -0.04611
0.01230 0.00114 0.00147

The idea here is to set a flag (the f variable) to 2 on the line matching z ~ and decrement its value by one each time we find a line matching ~ ~. Then, we print fields 4, 5 and 6 only on lines where f is 1 and which have at least 5 fields.

For both examples, to get pretty printing, you can use -vOFS="\t", or even better printf:

$ awk '/z ~/{f=2;} /~ ~/{f--}; (f==1 && NF>5){printf "%10s%10s%10s\n", $4, $5, $6} ' file
   0.00781   0.00108   0.00038
   0.01271  -0.01507   0.02839
  -0.05015  -0.01803   0.01588
   0.01733   0.03089  -0.04611
   0.01230   0.00114   0.00147
5
  • Note, check for next line existence would be reasonable (getline nl) > 0 Jun 6, 2017 at 12:25
  • @MarianKoniuszko you're welcome. If one of the answers here solved your issue, please take a moment and accept it by clicking on the check mark to the left. That will mark the question as answered and is the way thanks are expressed on the Stack Exchange sites.
    – terdon
    Jun 6, 2017 at 13:03
  • @RomanPerekhrest would it? Why? If there is no next line, then we don't need to print anything, the f will never be decremented and the script should exit gracefully enough.
    – terdon
    Jun 6, 2017 at 13:04
  • @terdon, not f. getline is used in your 1st approach. There's no f in it. But there is a loop with getline on each iteration Jun 6, 2017 at 13:08
  • 1
    @RomanPerekhrest ah, yes, sorry. OK, yes, if this is run on the last line for some reason, that would print empty fields. Fair enough.
    – terdon
    Jun 6, 2017 at 13:10
3

Working

  1. We first isolate the range as in-between /~ ~/ lines. Anything outside is deleted.
  2. The range outliers themselves are deleted as well.
  3. Now we have the proper lines to work on: In these we place the marker \n at the beginning of the 4th field and another one at the end of the 6th.
  4. Finally we strip anything outside of these markers and what remains are the 4th, 5th, and 6th fields + their intervening spaces (unmodified).

sed -ne '
   /~ ~/,//!d
   //d
   s/[^[:space:]]\{1,\}/&\n/6
   s/[^[:space:]]\{1,\}/\n&/4
   s/.*\n\(.*\)\n.*/\1/p
' yourfile

Results

0.00781      0.00108      0.00038
0.01271     -0.01507      0.02839
-0.05015     -0.01803      0.01588
0.01733      0.03089     -0.04611
0.01230      0.00114      0.00147
2

Try to break your operation into a few steps, each of which could be achieved by using different programs with their basic functionality.

For example, first find the pattern /z ~/ and print the next 6 lines (grep -A6 "z ~"), then print the 4th, 5th and 6th columns. Finally, filter out only those which have a digit in them, so that the line between the pattern match and digits are thrown away.

To wrap it in a command:

grep -A6 "z ~" file | awk '{ print $4, $5, $6 }' | grep -E "[[:digit:]]"`
1

As far as matching those lines, the /^~ [A-Z]/ pattern would be sufficient, and for each print the corresponding fields 4,5, and 6.

Awk version would be:

$ awk '/^~ [A-Z]/{printf("%-8s\t%-8s\t%-8s\n",$4,$5,$6)}' input.txt
0.00781     0.00108     0.00038 
0.01271     -0.01507    0.02839 
-0.05015    -0.01803    0.01588 
0.01733     0.03089     -0.04611
0.01230     0.00114     0.00147 

And the perl translation of the same thing:

$ perl -ane 'printf("%-8s\t%-8s\t%-8s\n",$F[3],$F[4],$F[5]) if /^~ [A-Z]/' input.txt                                     
0.00781     0.00108     0.00038 
0.01271     -0.01507    0.02839 
-0.05015    -0.01803    0.01588 
0.01733     0.03089     -0.04611
0.01230     0.00114     0.00147 

Notice that here we use printf() function with left-justified flags %-8s to make proper formatting.

Alternative way would be to treat the desired numbers as floating points ones, and use %f specifier instead of %-8s, but that adds extra zeroes to some of the numbers.

0

You may reconsider your logic. Instead of counting lines it seems more reliable here to completely base the extraction on the patterns of the lines themselves. Maybe this one is of use:

 awk '/^~ [A-Z]/' t.txt |cut -f 4-6 -d " "

Meaning: Extract only lines which start with "~" and where the 3rd character is a capital letter. Then feed the awk output through cut, extracting only fields 4 to 6 (start counting with 1) and setting delimiter to one whitespace.

1
  • I think the OP has tabs, not spaces. Either way, this won't work. If it's tabs, your cut gets the wrong characters but even if it's spaces, cut doesn't concatenate its field separators so two spaces and one space are not the same. You can see this if you actually try your solution on the OP's example.
    – terdon
    Jun 6, 2017 at 12:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .