I have an input file with this general structure. I only want to extract the values step and weight from the hill blocks and put them into a output file using awk/sed/grep. Hill blocks are arranged in similar fashion throughout in the input file.

Edit : I am using MAC OSX.

 configuration {
 step         5000
 dt 2.000000e+00

colvar {
name d1
x  1.70882305580118e+01
v  0.00000000000000e+00

1.85104129628346e-02 9.71380137561312e-02 4.00538287370335e-02
1.25662994200839e-02 9.88655406140091e-02 1.41657757894898e-01

hill {
step            0
weight    1.00000000000000e-01
centers   1.23563844380284e+02
widths    1.25331413731550e+00
 hill {
 step          100
 weight    1.00000000000000e-01
centers   1.19065310650377e+02
widths    1.25331413731550e+00

Through some other answers I manage to found some help :-

 sed 's/^.*weight//' diol_colvar.colvars.state > hill.txt
 sed 's/^.*step//' diol_colvar.colvars.state > hill.txt

Sadly this is not working as I wanted.

I want my output something like this :-

  0     1.00000000000000e-01
  100   1.00000000000000e-01

Please help me sort this issue.


  • from what I can understand, you want step and weight values only from hill blocks and put the values side by side? is that correct? – Sundeep Sep 3 '16 at 9:06
  • @sp asic yes, sorry if I dint make it clear. – Grayrigel Sep 3 '16 at 9:21
  • no issues.. but some more details will help in simpler solution... are step and weight are guaranteed to be next to each other? is there a chance of them appearing next to each other outside hill blocks? – Sundeep Sep 3 '16 at 9:23
  • @sp asic yes, all hill blocks are arranged in similar fashion and step and weight are always appearing similarly. – Grayrigel Sep 3 '16 at 9:31
  • alright, please add these details to question, while I will try to give an answer – Sundeep Sep 3 '16 at 9:39

1) With sed

Assuming step and weight occur in consecutive lines,

$ sed -nE '/step/{N;s/.*step\s+(\S+).*\n.*weight\s+(\S+).*/\1\t\2/p}' ip.txt 
0   1.00000000000000e-01
100 1.00000000000000e-01
  • -nE do not print lines by default and use extended regex
  • /step/ match if lines containing step
  • N to get the next line


The above was tested on GNU sed 4.2.2. The below might help on OS X and other versions. See this Q&A on SO for details, main point being \s might not work same as GNU sed

sed -nE '/step/{N;s/.*step[[:space:]]+([^[:space:]]+).*\n.*weight[[:space:]]+([^[:space:]]+).*/\1\t\2/p}' ip.txt

2) With awk

$ awk 'a ~ /step/ && /weight/{print v"\t"$2} {a=$0; v=$2}' ip.txt 
0   1.00000000000000e-01
100 1.00000000000000e-01
  • {a=$0; v=$2} saves the line and second field
  • a ~ /step/ && /weight/ match if previous line contains step and current line contains weight

Assuming the hill blocks are all similar to input given, we can match three consecutive lines to restrict the match only to hill blocks

awk 'b ~ /hill/ && a ~ /step/ && /weight/{print v"\t"$2} {b=a; a=$0; v=$2}' ip.txt

To save the results, add > output_filename to end of command


| improve this answer | |
  • awk works fine for me. Thanks a lot man, I wish I could have liked it (less reputation). However, sed solution shows this error (sed: 1: "/step/{N;s/.*step\s+(\S ...": bad flag in substitute command: '}' ) – Grayrigel Sep 3 '16 at 10:05
  • @VikasDubey, glad to hear it works.. see What should I do when someone answers my question? for next steps :) – Sundeep Sep 3 '16 at 10:06
  • @VikasDubey, what is your sed version? it works for me on GNU sed 4.2.2 – Sundeep Sep 3 '16 at 10:19
  • 1
    @vikasDubey, see the answer and ensuing comment by same author to stackoverflow.com/questions/12178924/…. sp asic's sed one-liner is correct, but does not apply to "OS X", because -E means extended regex, but not enhanced extended as OS X would need it to be. In OS X extended regex does not include the syntax \s to account for space as it normally does for GNU sed on a non-Apple GNU linux OS. So it's normal to see that sed one-liner fails for you. See what you get if you replace \s by [:space:] ... – Cbhihe Sep 4 '16 at 16:35
  • 1
    @Cbhihe, thanks for the pointer on OS X issue, updated the answer – Sundeep Sep 4 '16 at 17:01

More easy to use awk twice: first time to extract hill { } block and second to extract step/weight values.

awk '/hill *{/,/}/ {print}' \
   | awk '$1 == "step" { st = $2 }; $1 == "weight" { print st "\t" $2}'

This command will work only if weight comes after step but not necessary on consecutive lines.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.