3

I have this XML file (example)

<This is a line of text with a year=2020 month=12 in it
This line of text does not have a year or month in it
This year=2021 is the current year the current month=1
This is the year=2021 the month=2/>


<This is a line of text with a year=33020 month=12 in it
This line of text does not have a year or month in it
This year=33020 is the current year the current month=1
This is the year=33020 the month=2/>

Using the sed installation provided by my Linux distribution ( sed (GNU sed) 4.2.2) I search within this file with the following regexp:

 sed -En 'N;s/\<(This.*2020.*[\s\S\n]*?)\>/\1/gp' test2.txt

However, it captures only this string:

<This is a line of text with a year=2020 month=12 in it
This line of text does not have a year or month in it

But I try to capture the entire first paragraph between < and > that contains the pattern.

What am I doing wrong here?

9
  • 2
    Sed really isn't the best tool for something like this. Are you open to solutions using other tools?
    – terdon
    Aug 2 at 9:38
  • yeah sure but they need to be linux native
    – user63898
    Aug 2 at 9:43
  • 1
    Is all text in your file organized in these XML-tag-like structures? Is it in reality an XML file? Alternatively, are the paragraphs you are interested always separated by a blank newline?
    – AdminBee
    Aug 2 at 9:56
  • 2
    Please post well-formed XML that we may test answers on. The document fragments that you show are not real XML.
    – Kusalananda
    Aug 2 at 11:01
  • 3
    What am I doing wrong here? You're not using an XML parser. stackoverflow.com/a/1732454/864696 Aug 2 at 19:57
7

The reason this doesn't work as you expect is that < and > do not need to be escaped in regular expressions, they don't have any special meaning. However, \< and \> do have special meaning for GNU extended regular expressions (which you activate with -E): they match at word boundaries. \< matches the beginning of a word and \> the end. So \<(This isn't actually matching the <, it is matching the beginning of the word This. Similarly for the \> at the end. The GNU sed manual has an example which is almost exactly what you're after:

$ sed -En '/./{H;1h;$!d} ; x; s/(<This.*2020.*?>)/\1/p;' file
<This is a line of text with a year=2020 month=12 in it
This line of text does not have a year or month in it
This year=2021 is the current year the current month=1
This is the year=2021 the month=2/>

I find sed particularly ill-suited to this sort of task. I would use perl instead:

$ perl -000 -ne 'chomp;/<.*2020.*?>/s && print "$_\n"; exit' file
<This is a line of text with a year=2020 month=12 in it
This line of text does not have a year or month in it
This year=2021 is the current year the current month=1
This is the year=2021 the month=2/>

Here, we are using Perl in "paragraph mode" (-000) which means that a "line" is defined by two consecutive \n characters, by a blank line. The script will:

  • chomp: remove the trailing newline at the end of the "line" (paragraph).
  • /<.*2020.*?>/s && print "$_\n": if this "line" (paragraph) matches a < then 0 or more characters until 2020 and zero or more characters and then a >, then print this line appending a newline character (print "$_\n"). The s modifier to the match operator allows . to match newlines.

Another option is awk:

$ awk 'BEGIN{RS="\n\n"} /<.*2020.+?>/' file
<This is a line of text with a year=2020 month=12 in it
This line of text does not have a year or month in it
This year=2021 is the current year the current month=1
This is the year=2021 the month=2/>

We set the record separator RS to two consecutive newlines and then match using the same regex as above. Since in awk the default behavior when a match is found (or any other operation returns true) is to print the current record, this will print out what you need.

9
  • Note that this will pick up nodes that happen to have other attributes that have the value 2020 (or where 2020 is a substring of some value or attribute name). You most likely also want to include the node's tag name in the expression (This in the example).
    – Kusalananda
    Aug 2 at 11:26
  • 1
    When collecting all lines in hold space, you can do H;1h, so for the first line you overwrite the appended line with just the line.
    – Philippos
    Aug 2 at 11:29
  • 1
    @Kusalananda yes, but that isn't part of the question. I don't want to guess what the OP really wants to do, all I have to go on is what's in the question.
    – terdon
    Aug 2 at 12:05
  • 2
    @user63898 well yes, of course it will. Why would you use this with -i? Your question is showing an example file with two "paragraphs" and you say you only want the first one. Remember that we cannot magically guess what you want: you need to tell us.
    – terdon
    Aug 2 at 12:06
  • 1
    @Kusalananda I know. But this is what the OP is asking for.
    – terdon
    Aug 2 at 12:57
6

First, most text-processing tools such as sed or awk work on a line-by-line basis, so matching entire paragraphs takes a little bit of extra effort. It is possible, but it is also one of the reasons for the unexpected output you are seeing.

Second, your input looks like structured text due to the XML-tag delimiting characters. As such, it might be best processed using xmlstarlet or another dedicated tool. (Update: Since you confirmed this in a comment now, I would strongly recommend using xmlstarlet or a similar tool.)

That said, if your text looks like in your example, and you have an awk installation that accepts multi-character record separators (like GNU Awk), the following program should work:

awk -v RS="<|/>" '/2020/' input.txt

The RS variable, if consisting of more than one character, will be interpreted as regular expression, so either a < or a /> will be treated as "record separator", instead of the default \n. Therefore, any matching conditions will apply to the entire text between these tags, and not just individual lines.

Result:

This is a line of text with a year=2020 month=12 in it
This line of text does not have a year or month in it
This year=2021 is the current year the current month=1
This is the year=2021 the month=2

Note that the "tag-open" < and "tag-close" /> character combinations are stripped from the output because they are chosen as record separator. On the other hand, this means it will also work if the "paragraphs" are not separated by blank lines. (However, if there is "stray" text matching your pattern outside of such tags, it would also be matched.)

You would place the regular expression you are looking for inside the / ... / part of the program (just as in a sed adress statement). If you are looking for a fixed string, however, I would recommend

awk -v RS="<|/>" 'index($0,"2020")' input.txt

instead.

5
  • the awk is working but i need the sed feature that can search and replace in file using regexp groups that is the text that captured will be replaces with comment xml tags ( <!-- \1 --> )
    – user63898
    Aug 2 at 11:03
  • 1
    I see. If that is the case, please state this in your question from the beginning. In your case, since your actual question starts to deviate substantially from the posted one, please open a new question including all relevant conditions (i.e. actual, if possibly anonymized, XML content along with desired results). You may still want to accept one of the answers here that most closely did what you were asking for in the question originally.
    – AdminBee
    Aug 2 at 11:05
  • You may want to test this with an XML file having an attribute called os2020 and/or an attribute value that is 2020-01-01 or vision 2020 or whatever, you get what I mean. And you don't even look at the node's tag name.
    – Kusalananda
    Aug 2 at 11:28
  • @Kusalananda of course. Had I known that this was about true XML parsing, I would not have suggested that. But the answer does what the OP originally asked about, so I hink I will let it stand for the time being. The caveats I placed on top should alert any future reader about the shortcomings ...+
    – AdminBee
    Aug 2 at 12:40
  • 2
    @user63898 awk can do regex search and replace like sed using the sub(), gsub(), and (on GNU awk) gensub() functions. with capture groups. Processing XML, though, should be done with an xml parser (like xmlstarlet or xq, or with an xml parsing library for perl or python or whatever), not with regular expressions. Also worth looking at it xml2, which converts XML to a line-oriented format suitable for use with line-oriented tools like sed.
    – cas
    Aug 2 at 12:46
5

Assuming a well-formed XML document like this:

<root>
<thing  year="2019"
        month="1"
        day="1" />
<thing  year="2020"
        month="5"
        day="13" />
<thing  year="2021"
        month="7"
        day="3" />
</root>

You may extract a copy of each thing node that has the value 2020 in their year attribute using xmlstarlet like this:

$ xmlstarlet sel -t -c '//thing[@year = "2020"]' -nl file
<thing year="2020" month="5" day="13"/>

Note that whitespace within a node, between its attributes, is irrelevant to the contents of the document.

5
  • xmlstarlet is not installed by default on the servers
    – user63898
    Aug 2 at 11:20
  • 2
    @user63898 Then you may want to install it, especially since you're dealing with XML documents.
    – Kusalananda
    Aug 2 at 11:23
  • this is not option for me , thanks fo rthe help
    – user63898
    Aug 2 at 12:03
  • @user63898 May I ask whether your issue relates to a private or professional project?
    – Kusalananda
    Aug 2 at 12:48
  • 2
    @user63898 yes, of course, if you don't have a hacksaw then it's perfectly fine to use a hammer as a substitute. they're both tools, after all, and you already have the hammer.
    – cas
    Aug 2 at 12:49
0

Using Raku (formerly known as Perl_6)

Here are two answers, inspired by other answers in this thread. The first answer splits on paragraphs (inspired by @terdon and @AdminBee), then greps for the correct year:

raku -e 'slurp.split("\n\n").grep(/2020/).put;' 

Result:

<This is a line of text with a year=2020 month=12 in it
This line of text does not have a year or month in it
This year=2021 is the current year the current month=1
This is the year=2021 the month=2/>

According to one Larry Wall, Raku provides functions that make it really easy to do more from inside the language, lessening the reliance on specialized command-line switches. See "Trick #2" at:

https://www.nntp.perl.org/group/perl.perl6.users/2020/07/msg9004.html

The second approach is using Raku's comb routine, which takes a regex "matcher" and breaks text into elements outside the match (useful for further processing). As the Raku Docs describe it, comb: "Searches for $matcher in $input and returns a Seq of non-overlapping matches limited to at most $limit matches."

raku -e '.put for slurp.comb(/^^ "<This" .*? "/>" $$ / ).grep(/2020/);' 

Result:

<This is a line of text with a year=2020 month=12 in it
This line of text does not have a year or month in it
This year=2021 is the current year the current month=1
This is the year=2021 the month=2/>

The code above breaks before a ^^ start-of-line assertion and after a $$ end-of-line assertion. By default, the . dot wildcard matches whitespace in Raku (including newlines), so comb above can break the text into multi-line chunks (elements).

Far-and-away, the most satisfying results on authentic XML documents will be using a dedicated XML tool and/or library, such as Raku with the community-supported XML module:

https://github.com/raku-community-modules/XML
https://raku.org/

2
  • the problem i have only perl 5 installed , can't install new perl
    – user63898
    Aug 17 at 10:39
  • @user63898 understandable. Maybe my answer will be helpful for others then. Aug 17 at 19:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.