1

I'd like to find lines with h2 and p and get the content between the tags ...

<main>Nothing</main>
<h2>Hello</h2><p>World</p>
<h2>Bells</h2><p>Walls</p>
<h2>Jelly</h2><p>Minus</p>
<p>Fluff</p>

... into a tab separated list:

Hello World
Bells Walls
Jelly Minus

I currently use:

grep -E "<h2>(.*)<\/h2><p>(.*)<\/p>" | sed -E "s/<h2>(.*)<\/h2><p>(.*)<\/p>/\1 \2/"

However, I find it sort of annoying that I have to first grep <pattern> and then do sed <the same pattern>. Is it possible to do this using a single utility, using the pattern only once?

6
  • You could do all your string matching and substitution with python or Perl etc in ‘one action’. But it’s not really clear what you mean by a single action? One process? One thread? One program? – Joe Healey Apr 8 '18 at 19:45
  • Is that the actual file or just a part of a file or an example? If you want that exact result then you can use the following long-winded code and modify it to suit your needs: cat input | awk 'NR==2,NR==4' | sed 's/<h2>//' | sed 's/<\/h2><p>/ \t/' | sed 's/<\/p>//' > file2 It's one command but not one action although what you exactly want is not entirely clear. – Nasir Riley Apr 8 '18 at 20:02
  • grep is to search for strings and can't be used to replace strings. You'll need sed for the latter. – Nasir Riley Apr 8 '18 at 20:06
  • In answer to the modified post, yes you could do this all in one go. I would suggest making a little python script (or whatever language you prefer) and create a compiled regex with the re module. I’m not at my computer right now to have a go at it, so I’ll leave this as a comment for now. – Joe Healey Apr 8 '18 at 20:06
  • I was going to ask the very same question... Since this is for Bash script switching to perl/python for this isn't really worth it. @Joe I think the OP means one single command, and expressing the pattern only once (DRY principle). – xenoid Apr 8 '18 at 20:10
2

Using an XML parser is a really good idea, but if you can't use one for some reason (the file is not well-formed, you don't have any parsers installed, etc.), you can use PERL for this:

$ perl -ne 'if(/<h2>(.*?)<\/h2><p>(.*?)<\/p>/){print "$1\t$2\n"}' filename.ext
Hello   World
Bells   Walls
Jelly   Minus

I prefer using lazy matches so that I don't get unintended results:

test.txt

<h1>Nothing</h1>
<h2>Hello</h2><p>World</p><h2>Goodbye</h2><p>Earth</p>
<h2>Bells</h2><p>Walls</p>
<h2>Jelly</h2><p>Minus</p>
<h3>Zip</h3>

$ perl -ne 'if(/<h2>(.*?)<\/h2><p>(.*?)<\/p>/){print "$1\t$2\n"}' test.txt
Hello   World
Bells   Walls
Jelly   Minus
$ perl -ne 'if(/<h2>(.*)<\/h2><p>(.*)<\/p>/){print "$1\t$2\n"}' test.txt
Hello</h2><p>World</p><h2>Goodbye       Earth
Bells   Walls
Jelly   Minus

As you can see, only using a regular expression isn't going to get all the cases that a domain-specific tool will. If you're OK with that, then that's fine; just be aware that you may get inaccurate results if the input doesn't match your pattern exactly!

1
  • This is as short as you could hope for. Is it possible to make a generalised solution where you can have any number of fields tab separated without having to explicitly type $1\t$2...$n? – forthrin Apr 9 '18 at 7:21
1

The right way with xmlstarlet tool (for parsing xml/html data):

xmlstarlet sel -t -m '//h2' -v 'concat(., "'$'\t''", ./following-sibling::p)' -n file

The output:

Hello   World
Bells   Walls
Jelly   Minus
2
  • I get Extra content at the end of the document: at the first character of <h2>Hello</h2>.... I agree about XML parsing but XPath has a beginner's threshold. I'd be very interested in a lightweight XML extractor with a simple syntax, like find("a").href to get all links in a document. Is there such a tool that comes with most standard distros? – forthrin Apr 9 '18 at 7:29
  • @forthrin, there shouldn't be any errors if your tags are in the root(parent) tag. Python provides a flexible functional syntax when searching though xml/html, but it requires more code. But the main thing you should keep in mind that sed, grep and alike are NOT a good choice for xml/html processing – RomanPerekhrest Apr 9 '18 at 7:56
1

For the regex you use, which contains unquoted (), needs Extended Regex syntax (or replace each ( and ) with \( and \)). That's simple.

And, probably avoid a greedy matching too much using [^<] instead of a dot.

Of course, you can set a variable and play with quotes using only sed:

$ a='<h2>([^<]*)<\/h2><p>([^<]*)<\/p>'                                                                    
$ sed -nE '/'"$a"'/s/'"$a"'/\1 \2/p' infile

But it gets better as this could be simplified. Sed remembers the last regex used and a left side of s// (empty) is enougth.

$ sed -nE '/'"$a"'/s//\1 \2/p' infile

Or, without variable:

$ sed -nE '/<h2>([^<]*)<\/h2><p>([^<]*)<\/p>/s//\1 \2/p' infile
Hello World
Bells Walls
Jelly Minus
2
  • I like the brevity, but I get: "/<h2>([^<]*)<\/h2><p>([ ...": extra characters at the end of p command. BSD sed May 10, 2005. – forthrin Apr 9 '18 at 7:18
  • @forthrin There was a typo in the last expression, corrected. Tested with GNU sed, should also work with BSD sed. – Isaac Apr 9 '18 at 7:38
0

Possible solution via sed:

sed 's/<[^13>]*>/ /g' test | sed 's/<h[13]>.*<\/h[13]>//' <file>

 Hello  World
 Bells  Walls
 Jelly  Minus

Second sed just removes unnecessary tags (<h1> or <h3>).

Pattern explanation:

/<[^13>]*>/ / - search for any * symbols in text that begins with < and ends with >. But between the tags the symbols 1 or 3 must not (^) be present.

1
  • This works in this particular sample, but would struggle if other non-interesting content was introduced to the text. – forthrin Apr 9 '18 at 7:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.