Remove text between consecutive occurences of strings

Question

I need to remove all text between two given strings in a text file. The strings may be located on different lines. For instance, in the following text file

@article{ginsberg_lifespan_2018,
    title = {On the lifespan of three-dimensional abstract gravity water waves with vorticity},
    abstract = {test1
test2  abstract {NS}

test3},
    language = {en},
    urldate = {2018-12-05},
    author = {Ginsberg, Daniel},
    month = dec,
    year = {2018}
}

@article{higaki_two-dimensional_2017,
    title = {On the two-dimensional steady {Navier}-{Stokes} equations related to flows around a rotating obstacle},
    abstract = {We study the two-dimensional stationary Navier-Stokes equations with rotating effect in the whole space. The unique existence and the asymptotics of solutions are obtained without the smallness assumption on the rotation parameter.},
    journal = {arXiv:1703.07372 [math]},
    author = {Higaki, Mitsuo and Maekawa, Yasunori and Nakahara, Yuu},
    month = mar,
    year = {2017},
    note = {arXiv: 1703.07372},
    keywords = {Mathematics - Analysis of PDEs}
}

I want to remove everything between abstract = and a }, which is always at the end of a line, including these strings. That is I want the following output:

@article{ginsberg_lifespan_2018,
    title = {On the lifespan of three-dimensional abstract gravity water waves with vorticity},
    language = {en},
    urldate = {2018-12-05},
    author = {Ginsberg, Daniel},
    month = dec,
    year = {2018}
}

@article{higaki_two-dimensional_2017,
    title = {On the two-dimensional steady {Navier}-{Stokes} equations related to flows around a rotating obstacle},
    journal = {arXiv:1703.07372 [math]},
    author = {Higaki, Mitsuo and Maekawa, Yasunori and Nakahara, Yuu},
    month = mar,
    year = {2017},
    note = {arXiv: 1703.07372},
    keywords = {Mathematics - Analysis of PDEs}
}

I know this kind of questions were already asked and I tried the posted solutions. For example, I used

perl -0777 -pe 's/abstract = .*},\n/\n/gs'

but this removes the text between the first occurence of abstract = and the last occurence of },, not consecutive occurences. That is I get

@article{ginsberg_lifespan_2018,
    title = {On the lifespan of three-dimensional gravity water waves with vorticity},

    keywords = {Mathematics - Analysis of PDEs}
}

How can I correct this command to obtain the desired result?

Answer 1

$ sed '/abstract = .*},$/d; /abstract = /,/},$/d' <file
@article{ginsberg_lifespan_2018,
    title = {On the lifespan of three-dimensional abstract gravity water waves with vorticity},
    language = {en},
    urldate = {2018-12-05},
    author = {Ginsberg, Daniel},
    month = dec,
    year = {2018}
}

@article{higaki_two-dimensional_2017,
    title = {On the two-dimensional steady {Navier}-{Stokes} equations related to flows around a rotating obstacle},
    journal = {arXiv:1703.07372 [math]},
    author = {Higaki, Mitsuo and Maekawa, Yasunori and Nakahara, Yuu},
    month = mar,
    year = {2017},
    note = {arXiv: 1703.07372},
    keywords = {Mathematics - Analysis of PDEs}
}

This first tries to delete a complete single line abstract entry, and if that does not work, tries to delete a multi-line abstract entry. A multi-line entry is a set of lines from a line containing abstract = to the next line that ends with },.

Annotated sed script:

/abstract = .*},$/d    # delete complete abstract line, skip to next input line
/abstract = /,/},$/d   # delete multi-line abstract entry

If you need to be more specific with the start string, you could use, for example, ^[[:blank:]]*abstract instead of the abstract bit of those expressions. This would allow for only spaces or tabs to precede the abstract = on those lines.

Answer 2

The solution with sed (for example) is to convert each start and end strings to one character, so we can use regexps that avoid (negate) one character [^…].

Convert to one character (lets assume that % (start) and # (end) could not appear on your file, more later):

<<<infile sed 's/abstract =/%/g; s/},\n/#/g'

Then, we can select (and erase) from the first start (%) character to the first end (#) character that follows:

sed 's/%[^#]*#//g'

The [^#] is required to make the match non-greedy.

Since some delimiting characters might still exist, we need to restore them.

sed 's/%/abstract =/g; s/#/},\n/g'    # assuming GNU sed.

And, of course, all the above must be applied to the whole file as the patterns might appear on different lines. So, we capture the whole file in the hold space:

sed 'H;1h;$!d;g;'

In one complete command line:

 <infile sed 'H;1h;$!d;g;  s/abstract =/%/g; s/},\n/#/g;
                           s/%[^#]*#//g ;
                           s/%/abstract =/g; s/#/},\n/g'

If the characters selected could exist in the input file, we may choose some other delimiters that are clear that will not exist on your text files.

The characters (bytes) with value 01 and 02 which are called SOH (start of heading) and STX (start of text) in ASCII are "control characters" that are quite uncommon in text files. To use them, we better build a shell script:

 #!/bin/bash
 start=$'\1'
 end=$'\2'
 startpattern='abstract ='
 endpattern=$'},\\\n'         # The newline needs a `\` for sed to work.

 sed 'H;1h;$!d;g;
      s/'"$startpattern"'/'"$start"'/g;
      s/'"$endpattern"'/'"$end"'/g;
      s/'"$start"'[^'"$end"']*'"$end"'//g;
      s/'"$start"'/'"$startpattern"'/g;
      s/'"$end"'/'"$endpattern"'/g'  <infile

Answer 3

You are right, this or similar questions have been asked umpteen times in here. How far would

sed '/abstract.*{/ {:L; /}/{d; b;}; N; bL; }' file

get you? After matching abstract, it - if necassry - loops until the } found.

EDIT: Allowing for the modified request:

sed '/abstract.*{/ {:L; /},$/{d; b;}; N; bL; }' file

Answer 4

Another sed answer here, drawing from other answers, particularly this one by Isaac.

Matching everything from an opening sequence up to the first closing sequence (here: },new-line) would require non-greedy matches. Generally, sed does not have them, nor the ability to negate multi-character strings in a regular expression. You can find extended explanations on U&L about this topic, one example with thorough answers being this question.

(The (IMMO elegant) concise solution by Kusalananda overcomes this limitation by using address ranges. Here, though, I'm aiming for an alternative that won't remove the whole first and last matching lines).

As explained in Isaac's answer, you can convert your closing sequence to a single character and then use a character class which includes the negation of that character to non-greedily remove text up to the first (new) closing sequence.

If you also don't want to rely on the assumption that that character will not appear in the input text, you can use escaping and replace it with some special string.
This is one of the possible ways to do it (the choice of "special" characters being completely arbitrary), given your requirements:

1. Put the whole input in pattern space to let sed match text that spans over multiple lines
2. Replace every \ (our escape character) with \b - so that character sequences as \s will be impossible
3. Replace every # (our single-character closing sequence) with \s - so that any # will be impossibile
4. Replace any occurrence of the sequence },new-line with #
5. Remove any occurrence of: zero or more spaces + the literal "abstract" + one space + one equal sign + zero or more characters except for (stopping at the first) # + one final #
6. Put back the legitimate },new-line sequences
7. Revert \s to #
8. Revert \b to \

The corresponding command is:

sed ' H;1h;$!d;g;
      s/\\/\\b/g;
      s/#/\\s/g;
      s/},\n/#/g;
      s/[[:blank:]]*abstract[[:blank:]]=[^#]*#//g;
      s/#/},\n/g;
      s/\\s/#/g;
      s/\\b/\\/g;'

One of the answers to the aforementioned question provides a similar example on escaping. It differs from this one mainly in that it does not aim to match multi-line spans of text.

Answer 5

You were almost there with your Perl code, just needed a few tweaks :

 perl -0777pe 's/abstract = .*?\},\n/\n/msg'

The /s flag to enable . to match newlines and the .*? regex to not be greedy.