Extract multiple instances of text between two words inclusive of starting word but exclusive of ending word

Question

A PDB file contains numerous paragraphs of conformations of a protein.

Each conformation starts with the keyword ATOM and ends with keyword END.

I am trying to read the file in bash such that I read every line from ATOM until END, but I don't want to read the word END.

I want to do this for each conformations (paragraphs) and store each paragraph in an array.

The file somewhat looks like this:

ATOM line 1...

ATOM line 2...

ATOM line 3...

# More lines....

END

ATOM line 1...

ATOM line 2...

ATOM line 3...

# more lines...

END

An ATOM to END is one conformation.

I want to be able to read each conformation into an array, including the ATOM but excluding the END.

I can do read text between two keywords exclusive of both the words, but I don't know how to include the starting word, but exclude the end word.

Also reading each conformation into an array such that conf[0] = first conformation , conf[1] = second conformation so on and so forth doesn't work.

Code:

#!/bin/bash

filename='coor.pdb'
echo Start
i=0
while read line; do
    conf[$i]=$(sed -n '/ATOM/,/END/{//!p}') 
    i=i+1           
done < $filename
echo $conf[0] > first_frame.data

Answer 1

#!/bin/bash

filename='coor.pdb'
echo Start
i=1
input=false
while read -r line
do
    if [ "${line%% *}" == "ATOM" ]
    then
        input=true
    elif [ "${line%% *}" == "END" ]
    then
        ((i++))                 # increase variable i by 1 == (i+1)
        rm -f "${i}_frame.data" # remove ${i}_frame.data" if exist
        input=false             # stop output lines until next ATOM
    fi
    if $input                   # if var INPUT is true add line to ${i}_frame.data file
    then
        echo "$line" >> "${i}_frame.data"
    fi
done < "$filename"

For future some sed hints:

sed '/ATOM/,/END/!d;/END/d'
sed -n '/ATOM/{:;N;s/\nEND//;T;p}'

So you can do task:

nl -s'.frame.data' -b p"^END" coor.pdb | 
sed -n '/ATOM/{s/^/echo \"/;:;s/ \{6,\}//;N;s/END//;T;s/\n  */\">/p}' |
bash

Answer 2

Text processing in bash is slow. Pure-bash string manipulation is good for text you already have in variables, or for reading very small files. I suspect computational biology files won't usually be small, so use a tool like awk that has a small startup cost, but processes text much more quickly than bash.

Assuming you really just want to split your pdb file:

awk -v RS='\nEND\n' '{ fn="frame" NR ".pdb"; print > fn; close(fn) }' "$filename"

Get awk to use \nEND\n as the input record separator, instead of newline, then you can even use its record counter. The output record separator is still the default ORS="\n". (very nice suggestion by Costas. I tweaked it so END has to be at the start of a line, and added close to make sure it doesn't use a ton of file-descriptors on inputs with very many conformations.)

My original idea was:

awk 'BEGIN{i=0; fn="frame0.pdb"}
     !/^END/ { print > fn; }
     /^END/{ close(fn); fn="frame" ++i ".pdb"; }' \
     "$filename"

awk caches file-handles, so multiple print > fn don't result in close-reopen of the file. (close(fn) does that. It's there just for efficiency, so awk doesn't end up with boatloads of open files.)

The logic is: print every full line to the current filename. When you see an END line, move on to the next filename. If there isn't another line after the last END, then the new filename will never be written, and no vestigial last file will be created.

OTOH, if you want to do something with an array of line-blocks in memory:

# add a `!/^END/` condition to the concat block if you want to avoid a stray newline after each END
awk 'BEGIN{i=0}  
     !/^END/ { arr[i] = arr[i] $0 "\n"; }  # concat onto this array element
     /^END/ { i++; }
     END{for (k in arr) { print arr[k]; > ("frame" k ".pdb") } }' \
"$filename"

Then you have an awk array of lines to do with as you please in the END block. It has good regex functions.

---

Failed attempt at bash driving sed (nvm, fails because sed doesn't read one-byte-at-a-time the way shell read does):

i=0
while true; do
    outf="frame${i}.data";
    ##### DON'T USE THIS, sed READS TOO MUCH #####
    strace -o sed.tr sed '/^END/q42' > "$outf";  # strace to see that the 2nd sed invocation finds the file empty
    ret=$?;
    ((i++));
    if [[ $ret == 0 ]];then  # sed didn't see END before EOF
        [[ -s $outf ]] || rm -f "$outf";  # clean up empty last file
        break;
    elif [[ $ret != 42 ]]; then
        echo some other sed error;
        break;
    fi;
done < "$filename"