How can I delete everything between two markers in a file?

Question

I have a text in a text file, where I want everything that is between strings like \{{[} and {]}\} to be deleted - including these strings themselves. These two string can lie on different lines as well as on the same line. In either case, on the line on which the beginning \{{[} lies, I don't want the text before, i.e. left, of it to be deleted - and the same holds for the text after {]}\}.

Here is an example: Given a text file with the content

Bla Bla bla bla \{{[} more bla bla
even more bla bla bla bla. 

A lot of stuff might be here.

Bla bla {]}\} finally done.

Nonetheless, the \{{[} show {]}\} goes on.

the script should return another text file with the content

Bla Bla bla bla  finally done.

Nonetheless, the  goes on.

Unfortunately this simple-looking task turned out to be too difficult for me to do with sed. I'm happy with any solution in any language, as long as I don't have to install anything on my standard linux machine (C and some java is already installed).

Answer 1

With perl:

perl -0777 -pe 's/\Q\{{[}\E.*?\Q{]}\}\E//gs'

Note that the whole input is loaded in memory before being processed.

\Qsomething\E is for something to be treated as a literal string and not a regular expression.

To modify a regular file in-place, add the -i option:

perl -0777 -i -pe 's/\Q\{{[}\E.*?\Q{]}\}\E//gs' file.txt

With GNU awk or mawk:

awk -v 'RS=\\\\\\{\\{\\[}|\\{\\]}\\\\}' -v ORS= NR%2

There, we're defining the record separator as either of those beginning or end markers (only gawk and mawk support RS being a regexp here). But we need to escape the characters that are regexp operator (backslash, {, [) and also the backslash once more because it's special in arguments to -v (used for things like \n, \b...), hence the numerous backslashes.

Then all we need to do is print every other record. NR%2 would be 1 (true) for every odd record.

For both solutions, we're assuming the markers are matched and those sections not nested.

To modify the file in-place, with recent versions of GNU awk, add the -i inplace option.

Answer 2

sed   -e:t -e'y/\n/ /;/\\{{\[}/!b'               \
      -e:N -e'/\\{{\[.*{\]}\\}/!N'               \
           -e's/\(\\{{\[}\).*\n/\1/;tN'          \
           -e'y/ /\n/;s/\\{{\[}/& /;ts'          \
      -e:s -e's/\(\[} [^ ]*\)\({\]}\\}\)/\1 \2/' \
      -ets -e's/..... [^ ]* .....//;s/ //g;bt'   \
<<""
#Bla Bla {]}\} bla bla \{{[} more bla bla
#even more bla bla bla bla. \{{[} 
#
#A lot of stuff might be here.
#hashes are for stupid syntax color only
#Bla bla {]}\} finally {]}\} done.
#
#Nonetheless, the \{{[} show {]}\} goes \{{[} show {]}\} on.

---

#Bla Bla {]}\} bla bla  finally {]}\} done.
#
#Nonetheless, the  goes  on.

Here's a much better way, though. Far fewer substitutions, and those that are done are for a couple characters at a time rather than .* all of the time. Practically the only time .* is used is to clear the pattern space of the between space when the first occurring start is definitely paired with the first following end. All of the rest of the time sed simply Deletes as much as it has to to get to the next occurring delimiter. don taught me that.

sed -etD -e:t -e'/\\{{\[}/!b'  \
    -e's//\n /;h;D'       -e:D \
    -e'/^}/{H;x;s/\n.*\n.//;}' \
    -ett    -e's/{\]}\\}/\n}/' \
    -e'/\n/!{$!N;s//& /;}' -eD \
<<""
#Bla Bla {]}\} bla bla \{{[} more bla bla
#even more bla bla bla bla. \{{[} 
#
#A lot of stuff might be here.
#hashes are for stupid syntax color only
#Bla bla {]}\} finally {]}\} done.
#
#Nonetheless, the \{{[} show {]}\} goes \{{[} show {]}\} on.

---

#Bla Bla {]}\} bla bla  finally {]}\} done.
#
#Nonetheless, the  goes  on.

The RHS \newline escapes might need to be replaced with literal backslashed escaped newlines, though.

Here's a more generic version:

#!/usr/bin/sed -f
####replace everything between START and END
   #branch to :Kil if a successful substitution
   #has already occurred. this can only happen
   #if pattern space has been Deleted earlier
    t Kil
   #set a Ret :label so we can come back here
   #when we've cleared a START -> END occurrence
   #and check for another if need be
    :Ret
   #if no START, don't
    /START/!b
   #sigh. there is one. get to work. replace it
   #with a newline followed by an S and save
   #a copy then Delete up to our S marker.
    s||\
S|
    h;D
   #set the :Kil label. we'll come back here from now
   #on until we've definitely got END at the head of
   #pattern space.
    :Kil
   #do we? 
    /^E/{
       #if so, we'll append it to our earlier save
       #and slice out everything between the two newlines
       #we've managed to insert at just the right points        
        H;x
        s|\nS.*\nE||
    }
   #if we did just clear START -> END we should
   #branch back to :Ret and look for another START
    t Ret
   #pattern space didnt start w/ END, but is there even
   #one at all? if so replace it w/ a newline followed
   #by an E so we'll recognize it at the next :Kil
    s|END|\
E|
   #if that last was successful we'll have a newline
   #but if not it means we need to get the next line
   #if the last line we've got unmatched pairs and are
   #currently in a delete cycle anyway, but maybe we
   #should print up to our START marker in that case?
    /\n/!{
       #i guess so. now that i'm thinking about it
       #we'll swap into hold space, and Print it
        ${  x;P;d
        }
       #get next input line and add S after the delimiting
       #newline because we're still in START state. Delete
       #will handle everything up to our marker before we
       #branch back to :Kil at the top of the script
        N
        s||&S|
    }
   #now Delete will slice everything from head of pattern space
   #to the first occurring newline and loop back to top of script.
   #because we've definitely made successful substitutions if we
   #have a newline at all we'll test true and branch to :Kil 
   #to go again until we've definitely got ^E
    D

...without comments...

#!/usr/bin/sed -f
    t Kil
    :Ret
    /START/!b
    s||\
S|
    h;D
    :Kil
    /^E/{
        H;x
        s|\nS.*\nE||
    }
    t Ret
    s|END|\
E|
    /\n/!{
        ${  x;P;d
        }
        N
        s||&S|
    }
    D

I copied the commented version to my clipboard and did:

{ xsel; echo; } >se.sed
chmod +x se.sed
./se.sed <se.sed

---

#!/usr/bin/sed -f
####replace everything between
   #branch to :Kil if a successful substitution
   #has already occurred. this can only happen
   #if pattern space has been Deleted earlier
    t Kil
   #set a Ret :label so we can come back here
   #when we've cleared a  occurrence
   #and check for another if need be
    :Ret
   #if no  at the head of
   #pattern space.
    :Kil
   #do we?
    /^E/{
       #if so, we'll append it to our earlier save
       #and slice out everything between the two newlines
       #we've managed to insert at just the right points
        H;x
        s|\nS.*\nE||
    }
   #if we did just clear  we should
   #branch back to :Ret and look for another , but is there even
   #one at all? if so replace it w/ a newline followed
   #by an E so we'll recognize it at the next :Kil
    s|END|\
E|
   #if that last was successful we'll have a newline
   #but if not it means we need to get the next line
   #if the last line we've got unmatched pairs and are
   #currently in a delete cycle anyway, but maybe we
   #should print up to our

Answer 3

If your file is test.txt you may use:

sed ':a;N;$!ba;s/\n/ /g' test.txt|sed 's/\\{{\[}.*{\]}\\}//' 

the first sed removes all newlines, the second remove the text inside the tags.

I don't know if you need a more general solution