sed regex fails to capture the entire paragraph containing the pattern

Question

I have this XML file (example)

<This is a line of text with a year=2020 month=12 in it
This line of text does not have a year or month in it
This year=2021 is the current year the current month=1
This is the year=2021 the month=2/>


<This is a line of text with a year=33020 month=12 in it
This line of text does not have a year or month in it
This year=33020 is the current year the current month=1
This is the year=33020 the month=2/>

Using the sed installation provided by my Linux distribution ( sed (GNU sed) 4.2.2) I search within this file with the following regexp:

 sed -En 'N;s/\<(This.*2020.*[\s\S\n]*?)\>/\1/gp' test2.txt

However, it captures only this string:

<This is a line of text with a year=2020 month=12 in it
This line of text does not have a year or month in it

But I try to capture the entire first paragraph between < and > that contains the pattern.

What am I doing wrong here?

Answer 1

The reason this doesn't work as you expect is that < and > do not need to be escaped in regular expressions, they don't have any special meaning. However, \< and \> do have special meaning for GNU extended regular expressions (which you activate with -E): they match at word boundaries. \< matches the beginning of a word and \> the end. So \<(This isn't actually matching the <, it is matching the beginning of the word This. Similarly for the \> at the end. The GNU sed manual has an example which is almost exactly what you're after:

$ sed -En '/./{H;1h;$!d} ; x; s/(<This.*2020.*?>)/\1/p;' file
<This is a line of text with a year=2020 month=12 in it
This line of text does not have a year or month in it
This year=2021 is the current year the current month=1
This is the year=2021 the month=2/>

---

I find sed particularly ill-suited to this sort of task. I would use perl instead:

$ perl -000 -ne 'chomp;/<.*2020.*?>/s && print "$_\n"; exit' file
<This is a line of text with a year=2020 month=12 in it
This line of text does not have a year or month in it
This year=2021 is the current year the current month=1
This is the year=2021 the month=2/>

Here, we are using Perl in "paragraph mode" (-000) which means that a "line" is defined by two consecutive \n characters, by a blank line. The script will:

• chomp: remove the trailing newline at the end of the "line" (paragraph).
• /<.*2020.*?>/s && print "$_\n": if this "line" (paragraph) matches a < then 0 or more characters until 2020 and zero or more characters and then a >, then print this line appending a newline character (print "$_\n"). The s modifier to the match operator allows . to match newlines.

---

Another option is awk:

$ awk 'BEGIN{RS="\n\n"} /<.*2020.+?>/' file
<This is a line of text with a year=2020 month=12 in it
This line of text does not have a year or month in it
This year=2021 is the current year the current month=1
This is the year=2021 the month=2/>

We set the record separator RS to two consecutive newlines and then match using the same regex as above. Since in awk the default behavior when a match is found (or any other operation returns true) is to print the current record, this will print out what you need.

Answer 2

First, most text-processing tools such as sed or awk work on a line-by-line basis, so matching entire paragraphs takes a little bit of extra effort. It is possible, but it is also one of the reasons for the unexpected output you are seeing.

Second, your input looks like structured text due to the XML-tag delimiting characters. As such, it might be best processed using xmlstarlet or another dedicated tool. (Update: Since you confirmed this in a comment now, I would strongly recommend using xmlstarlet or a similar tool.)

That said, if your text looks like in your example, and you have an awk installation that accepts multi-character record separators (like GNU Awk), the following program should work:

awk -v RS="<|/>" '/2020/' input.txt

The RS variable, if consisting of more than one character, will be interpreted as regular expression, so either a < or a /> will be treated as "record separator", instead of the default \n. Therefore, any matching conditions will apply to the entire text between these tags, and not just individual lines.

Result:

This is a line of text with a year=2020 month=12 in it
This line of text does not have a year or month in it
This year=2021 is the current year the current month=1
This is the year=2021 the month=2

Note that the "tag-open" < and "tag-close" /> character combinations are stripped from the output because they are chosen as record separator. On the other hand, this means it will also work if the "paragraphs" are not separated by blank lines. (However, if there is "stray" text matching your pattern outside of such tags, it would also be matched.)

You would place the regular expression you are looking for inside the / ... / part of the program (just as in a sed adress statement). If you are looking for a fixed string, however, I would recommend

awk -v RS="<|/>" 'index($0,"2020")' input.txt

instead.

Answer 3

Assuming a well-formed XML document like this:

<root>
<thing  year="2019"
        month="1"
        day="1" />
<thing  year="2020"
        month="5"
        day="13" />
<thing  year="2021"
        month="7"
        day="3" />
</root>

You may extract a copy of each thing node that has the value 2020 in their year attribute using xmlstarlet like this:

$ xmlstarlet sel -t -c '//thing[@year = "2020"]' -nl file
<thing year="2020" month="5" day="13"/>

Note that whitespace within a node, between its attributes, is irrelevant to the contents of the document.

Answer 4

Using Raku (formerly known as Perl_6)

Here are two answers, inspired by other answers in this thread. The first answer splits on paragraphs (inspired by @terdon and @AdminBee), then greps for the correct year:

raku -e 'slurp.split("\n\n").grep(/2020/).put;' 

Result:

<This is a line of text with a year=2020 month=12 in it
This line of text does not have a year or month in it
This year=2021 is the current year the current month=1
This is the year=2021 the month=2/>

According to one Larry Wall, Raku provides functions that make it really easy to do more from inside the language, lessening the reliance on specialized command-line switches. See "Trick #2" at:

https://www.nntp.perl.org/group/perl.perl6.users/2020/07/msg9004.html

The second approach is using Raku's comb routine, which takes a regex "matcher" and breaks text into elements outside the match (useful for further processing). As the Raku Docs describe it, comb: "Searches for $matcher in $input and returns a Seq of non-overlapping matches limited to at most $limit matches."

raku -e '.put for slurp.comb(/^^ "<This" .*? "/>" $$ / ).grep(/2020/);' 

Result:

<This is a line of text with a year=2020 month=12 in it
This line of text does not have a year or month in it
This year=2021 is the current year the current month=1
This is the year=2021 the month=2/>

The code above breaks before a ^^ start-of-line assertion and after a $$ end-of-line assertion. By default, the . dot wildcard matches whitespace in Raku (including newlines), so comb above can break the text into multi-line chunks (elements).

Far-and-away, the most satisfying results on authentic XML documents will be using a dedicated XML tool and/or library, such as Raku with the community-supported XML module:

https://github.com/raku-community-modules/XML
https://raku.org/