4

I have a text in a text file, where I want everything that is between strings like \{{[} and {]}\} to be deleted - including these strings themselves. These two string can lie on different lines as well as on the same line. In either case, on the line on which the beginning \{{[} lies, I don't want the text before, i.e. left, of it to be deleted - and the same holds for the text after {]}\}.

Here is an example: Given a text file with the content

Bla Bla bla bla \{{[} more bla bla
even more bla bla bla bla. 

A lot of stuff might be here.

Bla bla {]}\} finally done.

Nonetheless, the \{{[} show {]}\} goes on.

the script should return another text file with the content

Bla Bla bla bla  finally done.

Nonetheless, the  goes on.

Unfortunately this simple-looking task turned out to be too difficult for me to do with sed. I'm happy with any solution in any language, as long as I don't have to install anything on my standard linux machine (C and some java is already installed).

  • @don_crissti yes, they could also lie on the same line. both cases are possible. I edited the question accordingly. – l7ll7 Nov 4 '15 at 17:16
5

With perl:

perl -0777 -pe 's/\Q\{{[}\E.*?\Q{]}\}\E//gs'

Note that the whole input is loaded in memory before being processed.

\Qsomething\E is for something to be treated as a literal string and not a regular expression.

To modify a regular file in-place, add the -i option:

perl -0777 -i -pe 's/\Q\{{[}\E.*?\Q{]}\}\E//gs' file.txt

With GNU awk or mawk:

awk -v 'RS=\\\\\\{\\{\\[}|\\{\\]}\\\\}' -v ORS= NR%2

There, we're defining the record separator as either of those beginning or end markers (only gawk and mawk support RS being a regexp here). But we need to escape the characters that are regexp operator (backslash, {, [) and also the backslash once more because it's special in arguments to -v (used for things like \n, \b...), hence the numerous backslashes.

Then all we need to do is print every other record. NR%2 would be 1 (true) for every odd record.

For both solutions, we're assuming the markers are matched and those sections not nested.

To modify the file in-place, with recent versions of GNU awk, add the -i inplace option.

  • Why are you escaping that many \ (backslashes)? The following line produces the same output for me: awk -v 'RS=\\\{\{\[\}|\{\]\}\\\}' -v ORS= NR%2 filename Also, can you explain the ORS= NR%2 part? – Kira Nov 4 '15 at 20:12
  • @Kira, that wouldn't work with gawk. You need 2 levels of backslash escaping, one for -v escape sequences and one for regular expressions. For -v, \{ is not a valid escape sequences so the result is unspecified and varies. mawk expands it to \{, gawk to { (the recent versions also issue a warning). In mawk, \\\{ expands to \\{. It only works in current versions of mawk because mawk still doesn't support the { regexp operator. – Stéphane Chazelas Nov 5 '15 at 10:50
  • Could you please tell me also, how I can input and output this to a file ? If my text is in file.txt, then $ cat file.txt | <<your code>> > file_new.txt doesn't work.... – l7ll7 Nov 5 '15 at 11:28
  • @user10324, that and <<my code>> < file.txt > file_new.txt should work. See also edit for the -i option in perl. – Stéphane Chazelas Nov 5 '15 at 11:51
3
sed   -e:t -e'y/\n/ /;/\\{{\[}/!b'               \
      -e:N -e'/\\{{\[.*{\]}\\}/!N'               \
           -e's/\(\\{{\[}\).*\n/\1/;tN'          \
           -e'y/ /\n/;s/\\{{\[}/& /;ts'          \
      -e:s -e's/\(\[} [^ ]*\)\({\]}\\}\)/\1 \2/' \
      -ets -e's/..... [^ ]* .....//;s/ //g;bt'   \
<<""
#Bla Bla {]}\} bla bla \{{[} more bla bla
#even more bla bla bla bla. \{{[} 
#
#A lot of stuff might be here.
#hashes are for stupid syntax color only
#Bla bla {]}\} finally {]}\} done.
#
#Nonetheless, the \{{[} show {]}\} goes \{{[} show {]}\} on.

#Bla Bla {]}\} bla bla  finally {]}\} done.
#
#Nonetheless, the  goes  on.

Here's a much better way, though. Far fewer substitutions, and those that are done are for a couple characters at a time rather than .* all of the time. Practically the only time .* is used is to clear the pattern space of the between space when the first occurring start is definitely paired with the first following end. All of the rest of the time sed simply Deletes as much as it has to to get to the next occurring delimiter. don taught me that.

sed -etD -e:t -e'/\\{{\[}/!b'  \
    -e's//\n /;h;D'       -e:D \
    -e'/^}/{H;x;s/\n.*\n.//;}' \
    -ett    -e's/{\]}\\}/\n}/' \
    -e'/\n/!{$!N;s//& /;}' -eD \
<<""
#Bla Bla {]}\} bla bla \{{[} more bla bla
#even more bla bla bla bla. \{{[} 
#
#A lot of stuff might be here.
#hashes are for stupid syntax color only
#Bla bla {]}\} finally {]}\} done.
#
#Nonetheless, the \{{[} show {]}\} goes \{{[} show {]}\} on.

#Bla Bla {]}\} bla bla  finally {]}\} done.
#
#Nonetheless, the  goes  on.

The RHS \newline escapes might need to be replaced with literal backslashed escaped newlines, though.

Here's a more generic version:

#!/usr/bin/sed -f
####replace everything between START and END
   #branch to :Kil if a successful substitution
   #has already occurred. this can only happen
   #if pattern space has been Deleted earlier
    t Kil
   #set a Ret :label so we can come back here
   #when we've cleared a START -> END occurrence
   #and check for another if need be
    :Ret
   #if no START, don't
    /START/!b
   #sigh. there is one. get to work. replace it
   #with a newline followed by an S and save
   #a copy then Delete up to our S marker.
    s||\
S|
    h;D
   #set the :Kil label. we'll come back here from now
   #on until we've definitely got END at the head of
   #pattern space.
    :Kil
   #do we? 
    /^E/{
       #if so, we'll append it to our earlier save
       #and slice out everything between the two newlines
       #we've managed to insert at just the right points        
        H;x
        s|\nS.*\nE||
    }
   #if we did just clear START -> END we should
   #branch back to :Ret and look for another START
    t Ret
   #pattern space didnt start w/ END, but is there even
   #one at all? if so replace it w/ a newline followed
   #by an E so we'll recognize it at the next :Kil
    s|END|\
E|
   #if that last was successful we'll have a newline
   #but if not it means we need to get the next line
   #if the last line we've got unmatched pairs and are
   #currently in a delete cycle anyway, but maybe we
   #should print up to our START marker in that case?
    /\n/!{
       #i guess so. now that i'm thinking about it
       #we'll swap into hold space, and Print it
        ${  x;P;d
        }
       #get next input line and add S after the delimiting
       #newline because we're still in START state. Delete
       #will handle everything up to our marker before we
       #branch back to :Kil at the top of the script
        N
        s||&S|
    }
   #now Delete will slice everything from head of pattern space
   #to the first occurring newline and loop back to top of script.
   #because we've definitely made successful substitutions if we
   #have a newline at all we'll test true and branch to :Kil 
   #to go again until we've definitely got ^E
    D

...without comments...

#!/usr/bin/sed -f
    t Kil
    :Ret
    /START/!b
    s||\
S|
    h;D
    :Kil
    /^E/{
        H;x
        s|\nS.*\nE||
    }
    t Ret
    s|END|\
E|
    /\n/!{
        ${  x;P;d
        }
        N
        s||&S|
    }
    D

I copied the commented version to my clipboard and did:

{ xsel; echo; } >se.sed
chmod +x se.sed
./se.sed <se.sed

#!/usr/bin/sed -f
####replace everything between
   #branch to :Kil if a successful substitution
   #has already occurred. this can only happen
   #if pattern space has been Deleted earlier
    t Kil
   #set a Ret :label so we can come back here
   #when we've cleared a  occurrence
   #and check for another if need be
    :Ret
   #if no  at the head of
   #pattern space.
    :Kil
   #do we?
    /^E/{
       #if so, we'll append it to our earlier save
       #and slice out everything between the two newlines
       #we've managed to insert at just the right points
        H;x
        s|\nS.*\nE||
    }
   #if we did just clear  we should
   #branch back to :Ret and look for another , but is there even
   #one at all? if so replace it w/ a newline followed
   #by an E so we'll recognize it at the next :Kil
    s|END|\
E|
   #if that last was successful we'll have a newline
   #but if not it means we need to get the next line
   #if the last line we've got unmatched pairs and are
   #currently in a delete cycle anyway, but maybe we
   #should print up to our
  • @don_crissti - ok. – mikeserv Nov 5 '15 at 0:03
  • 2
    definite +1 for getting sed to do this. the code, without comments, looks amazingly cryptic :) – l7ll7 Nov 5 '15 at 11:08
0

If your file is test.txt you may use:

sed ':a;N;$!ba;s/\n/ /g' test.txt|sed 's/\\{{\[}.*{\]}\\}//' 

the first sed removes all newlines, the second remove the text inside the tags.

I don't know if you need a more general solution

  • I tested this approach, is there some way to preserve the line breaks ? A more general input in my question was edited, to make clear, what I want. – l7ll7 Nov 4 '15 at 17:22
  • That assumes GNU sed and that there's only one such section to remove in the input (as that will remove anything from the first occurrence of the begin marker to the last occurrence of the end marker). – Stéphane Chazelas Nov 4 '15 at 17:22
  • @StéphaneChazelas So, you mean it is not feasable to do it ? – l7ll7 Nov 4 '15 at 17:26
  • @user10324, it's doable in sed (at least with the GNU implementation), but a lot easier with perl or gawk which will also be installed on your standard Linux machine. – Stéphane Chazelas Nov 4 '15 at 17:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.