Text between two tags

Question

I want to retrieve whatever is between these two tags – <tr> </tr> – from an html doc. Now I don't have any specific html requirements that would warrant for an html parser. I just plain need something that matches <tr> and </tr> and gets everything in between and there could be multiple trs. I tried awk, which works, but for some reason it ends up giving me duplicates of each row extracted.

awk '
/<TR/{p=1; s=$0}
p && /<\/TR>/{print $0 FS s; s=""; p=0}
p' htmlfile> newfile

How to go about this?

Answer 1

If you only want ... of all <tr>...</tr> do:

grep -o '<tr>.*</tr>' HTMLFILE | sed 's/\(<tr>\|<\/tr>\)//g' > NEWFILE

For multiline do:

tr "\n" "|" < HTMLFILE | grep -o '<tr>.*</tr>' | sed 's/\(<tr>\|<\/tr>\)//g;s/|/\n/g' > NEWFILE

Check the HTMLFILE first of the char "|" (not usual, but possible) and if it exists, change to one which doesn't exist.

Answer 2

You do have a requirement that warrants an HTML parser: you need to parse HTML. Perl's HTML::TreeBuilder, Python's BeautifulSoup and others are easy to use, easier than writing complex and brittle regular expressions.

perl -MHTML::TreeBuilder -le '
    $html = HTML::TreeBuilder->new_from_file($ARGV[0]) or die $!;
    foreach ($html->look_down(_tag => "tr")) {
        print map {$_->as_HTML()} $_->content_list();
    }
' input.html

or

python -c 'if True:
    import sys, BeautifulSoup
    html = BeautifulSoup.BeautifulSoup(open(sys.argv[1]).read())
    for tr in html.findAll("tr"):
        print "".join(tr.contents)
' input.html

Answer 3

sed and awk are not well suited for this task, you should rather use a proper html parser. For example hxselect from w3.org:

<htmlfile hxselect -s '\n' -c 'tr'

Answer 4

If ruby is available you can do the following

ruby -e 'puts readlines.join[/(?<=<tr>).+(?=<\/tr>)/m].gsub(/<\/?tr>/, "")' file

where file is your input html file. The command executes a Ruby one-liner. First, it reads all lines from file and joins them to a string, readlines.join. Then, from the string it selects anything between (but not including) <tr> and <\/tr> that is one character or longer irrespective of newlines, [/(?<=<tr>).+(?=<\/tr>)/m]. Then, it removes any <tr> or </tr> from the string, gsub(/<\/?tr>/, "") (this is necessary to handle nested tr tags). Finally, it prints the string, puts.

You said that a html parser is not warranted for you but it is very easy to use Nokogiri with ruby and it makes the command simpler.

ruby -rnokogiri -e 'puts Nokogiri::HTML(readlines.join).xpath("//tr").map { |e| e.content }' file

-rnokogiri loads Nokogiri. Nokogiri::HTML(readlines.join) reads all lines of file. xpath("//tr") picks out every tr element and map { |e| e.content } picks out the content for each element, i.e. what is between <tr> and </tr>.

Answer 5

pup

Example using pup (which uses CSS selectors):

pup -f myfile.html tr

To print only text without tags, use: pup -f myfile.html tr text{}.

Here are few examples with curl:

curl -sL https://www.iana.org/ | pup tr text{}
pup -f <(curl -sL https://www.iana.org/) tr text{}

xpup

Example using xpup for HTML/XML parsing (which supports XPath):

xpup -f myfile.html "//tr"

Answer 6

grep

To retrieve content within tr tag across multiple lines, pass it through xargs first, for example:

curl -sL https://www.iana.org/ | xargs | egrep -o "<tr>.*?</tr>"

To return only inner HTML, use:

curl -sL https://www.iana.org/ | xargs | grep -Po "<tr>\K(.*?)</tr>" | sed "s/..tr.//g"

^{Check the syntax for perlre extended patterns.}

^{Note: For quicker performance, you may consider ripgrep which has similar syntax.}

Answer 7

if it is just a quick listing of <tr>s this could help:

perl -ne 'print if /<tr>/../</tr>/' your.html > TRs.log

cheers