How to remove text between two keywords in a textfile

Question

Context : GNU/Linux Ubuntu.

I've a file made of thousands of lines and I'd like to have a script that will remove some lines between 2 specific keywords.
Initial file is like:

bla bla
...
bla bla
keyword1
bla bla
...
bla bla
keyword2
bla bla
...
bla bla

I would like to keep all file except the part between keyword1 and keyword2.

Let's consider that keyword1 and keyword2 appear once and only once in the file; these keywords may be preceded or followed by other characters on their lines like spaces or < or >

The lines with the keywords on them look like this (they are actually XML-based files):

<keyword2>  

The keywords can be kept in the file or be removed along with the enclosed text, I'm fine with both outcomes.

I did not find out how to proceed using grep. I'm not familiar with awk; could it make it?

Answer 1

I put your sample text into the file file, and tested also with <> around the keywords.

This command with sed removes the keywords

$ < file sed '/keyword1/,/keyword2/d'
bla bla
...
bla bla
bla bla
...
bla bla

This command with sed preserves the keywords

$ < file sed -n -e '1,/keyword1/p' -e '/keyword2/,$p'
bla bla
...
bla bla
<keyword1>
<keyword2>
bla bla
...
bla bla

Answer 2

Using Raku (formerly known as Perl_6)

raku -ne '.put unless /keyword1/ ^fff^ /keyword2/;'

Sample Input:

bla bla
...
bla bla
keyword1
bla bla
...
bla bla
keyword2
bla bla
...
bla bla

Sample Output:

bla bla
...
bla bla
keyword1
keyword2
bla bla
...
bla bla

Briefly, Raku's -ne command line flags tell Raku to execute code without autoprinting. Printing is accomplished via first instruction .put ('print-using-terminator', i.e. newline). The . dot preceding put is short for $_.put wherein $_ represents the topic variable (in this case, containing data from the input line).

The fff instruction is Raku's sed-like "flip-flop" operator, which turns ON/OFF based upon the two surrounding regexes. In Raku (and Perl5), unless is an alias to if not. Finally, the ^ carets surrounding fff to give ^fff^ tell Raku to exclude the endpoints.

Because unless is a negation, ^fff^ negates the exclusion of endpoints, which preserves keyword1 and keyword2 in the output. Using fff instead of ^fff^ deletes keyword1 and keyword2 from the output.

(Note here, that if you really want to parse an XML file, then one-liner Raku solutions can be crafted using Raku's XML module).

https://unix.stackexchange.com/search?q=Raku+%5BXML%5D
https://github.com/raku-community-modules/XML
https://raku.org

Answer 3

The previous suggestion about sed's won't give expected results when the "keyword" is not the only word in the line. If you want to extract a text between any two words from any paragraph regardless of their positions, you will need Perl, particularly Perl's File Slurp

For example, let's say we have this text:

Sir Arthur Conan Doyle was born on May 22, 1859, in Edinburgh. 
He studied medicine at the University of Edinburgh and began to write stories while he was a student. 
Over his life he produced more than 30 books, 150 short stories, poems, plays, and essays across a wide range
of genres. 
His most famous creation is the detective Sherlock Holmes, who he introduced in his first novel, A Study in Scarlet (1887). 
This was followed in 1889 by an historical novel, Micah Clarke.

My keywords here are "medicine" and "Holmes" respectively.

The results with sed will delete exactly the first and the last lines from the paragraph. While the expected result should also remove the part of the sentence that is before and including medicine, plus after and including Holmes.

Let's try Perl's File Slurp:

perl -0777 -i -pe 'push @a,/medicine(.*?)Holmes/s;END{print "@a"}' myparagraph.txt

The output:

at the University of Edinburgh and began to write stories while he was a student. 
Over his life he produced more than 30 books, 150 short stories, poems, plays, and essays across a wide range
of genres. 
His most famous creation is the detective Sherlock 

Answer 4

As we don't have a real XML document to work with, I'm going to assume that the document in question looks somewhat like the following:

<?xml version="1.0"?>
<root>
  <entry>
    <name>Joe</name>
    <number>133</number>
  </entry>
  <entry>
    <name>Mary</name>
    <number>123</number>
  </entry>
  <entry>
    <name>Stan</name>
    <number>233</number>
  </entry>
</root>

The task is also a bit unclear, so I will show how to

1. Delete one of the entry nodes given a name value.
2. Change the number value for one entry node, given a name value.
3. Remove the contents of one of the entry nodes, given a name value.

This will first be done using the reasonably common command-line XML parser xmlstarlet and then the lesser-known xq (from https://kislyuk.github.io/yq/), a wrapper around the well-known JSON parser jq.

First, using XPath syntax with xmlstarlet:

1. Deleting Stan:

   xmlstarlet ed \
       --var name '"Stan"' \
       --delete '//entry[name = $name]' file.xml
   

   This takes the XPath string "Stan", assigns it to the internal variable $name, and uses it to pick out the entry node that has that particular name value. The entry node may be located anywhere in the document since we use //entry rather than a specific path such as /root/entry to find it.

   The found node is deleted by xmlstarlet, and the resulting XML document is written to standard output.

   The resulting document:

   <?xml version="1.0"?>
   <root>
     <entry>
       <name>Joe</name>
       <number>133</number>
     </entry>
     <entry>
       <name>Mary</name>
       <number>123</number>
     </entry>
   </root>
   

2. Changing Stan's number to 455:

   xmlstarlet ed \
       --var name '"Stan"' \
       --var value '455' \
       --update '//entry[name = $name]/number' \
       --expr '$value' file.xml
   

   This is similar to the first command in that it selects the entry node that we're interested in using the internal variable $name, containing an XPath string. Rather than deleting the found node(s), it updates the number sub-node using the provided value in the internal $value variable.

   The resulting document:

   <?xml version="1.0"?>
   <root>
     <entry>
       <name>Joe</name>
       <number>133</number>
     </entry>
     <entry>
       <name>Mary</name>
       <number>123</number>
     </entry>
     <entry>
       <name>Stan</name>
       <number>455</number>
     </entry>
   </root>
   

3. Emptying Stan's record:

   xmlstarlet ed \
       --var name '"Stan"' \
       --update '//entry[name = $name]' \
       --value '' file.xml
   

   Again, this shows that we may empty a node by updating its value to an empty string.

   The resulting document:

   <?xml version="1.0"?>
   <root>
     <entry>
       <name>Joe</name>
       <number>133</number>
     </entry>
     <entry>
       <name>Mary</name>
       <number>123</number>
     </entry>
     <entry/>
   </root>
   

The xq wrapper around jq parses and transcodes the XML document into JSON. It then applies a jq expression to the generated JSON document and optionally translates it back to XML.

Given the document at the start of this answer, even though the input is an XML document, xq will internally work with a JSON document that looks like this:

{
  "root": {
    "entry": [
      {
        "name": "Joe",
        "number": "133"
      },
      {
        "name": "Mary",
        "number": "123"
      },
      {
        "name": "Stan",
        "number": "233"
      }
    ]
  }
}

1. Deleting Stan:

   xq --xml-output \
       --arg name 'Stan' \
       'del(.root.entry[] | select(.name == $name))' file.xml
   

   This uses the del() function of jq to delete the given path. The path is found by selecting the element(s) from the .root.entry array with a .name key with the value given by $name, an internal variable that we set on the command line.

2. Changing Stan's number to 455:

   xq --xml-output \
       --arg name 'Stan' \
       --arg value 455 \
       '(.root.entry[] | select(.name == $name)).number |= $value' file.xml
   

   This is similar to the previous expression, but rather than deleting the selected node(s) with del(), we access the .number key and update its value using the internal variable $value.

3. Emptying Stan's record:

   xq --xml-output \
       --arg name 'Stan' \
       '(.root.entry[] | select(.name == $name)) |= null' file.xml
   

   Again, we use a similar expression that selects the node(s) we're interested in and then updates it with null to empty it. Using empty in place of null would delete the node, so that's another way of achieving the same result as under the first point just above.

The main difference between these xmlstarlet and xq/jq expressions is that we're using an absolute path with xq, while we use // in the XPath expression with xmlstarlet to search for the nodes we're interested in recursively. You may do recursive searching with xq as well, but it's a bit more tricky, and the example we chose to work with here doesn't require it.