Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I wanna retrieve whatever is between these two tags – <tr> </tr> – from an html doc. Now I don't have any specific html requirements that would warrant for an html parser. I just plain need something that matches <tr> and </tr> and gets everything in between and there could be multiple trs. I tried awk, which works, but for some reason it ends up giving me duplicates of each row extracted.

awk '
/<TR/{p=1; s=$0}
p && /<\/TR>/{print $0 FS s; s=""; p=0}
p' htmlfile> newfile

How to go about this?

share|improve this question
    
IIUC your awk script should be: '/<tr/{p=1}; p; /<\/tr>/{p=0}'. Post some example input and expected output if it doesn't work. –  Thor Feb 13 '13 at 14:23

4 Answers 4

up vote 1 down vote accepted

if you only want ... of all <tr>...</tr> do:
grep -o '<tr>.*</tr>' HTMLFILE | sed 's/\(<tr>\|<\/tr>\)//g' > NEWFILE

for multiline do:
cat HTMLFILE | tr "\n" "|" | grep -o '<tr>.*</tr>' | sed 's/\(<tr>\|<\/tr>\)//g' | sed 's/|/\n/g' > NEWFILE
check the HTMLFILE first of the char "|" (not usual, but possible) and if it exists, change to one which doesn't exist.

share|improve this answer
    
That'll only work if the start and end tags are on the same line. –  l0b0 Feb 13 '13 at 12:01
    
echo "bla<tr>foo</tr>bla<tr>bar</tr>bla" | grep -o '<tr>.*</tr>' | sed 's/\(<tr>\|<\/tr>\)//g' gives fooblabar. The bla should not be there? –  N.N. Feb 13 '13 at 12:03
    
@l0b0 correct. will go for a multiline-compatible one... –  xx4h Feb 13 '13 at 12:07
    
grep -Po '<tr>.*?</tr>' would return one result per line in @N.N.'s case, but it's not portable. –  l0b0 Feb 13 '13 at 12:15
    
I'm not sure what you mean by 'specs' or 'spec-style' but note that your web browser uses an html parser and an html parser will parse html regardless of how it is written. It will not parse things that are not html, but then, neither will your browser, so no one would bother writing "html" that a parser cannot parse. In other words: A decent parser is absolutely certainly your best bet for doing this. –  goldilocks Feb 13 '13 at 12:38

You do have a requirement that warrants an HTML parser: you need to parse HTML. Perl's HTML::TreeBuilder, Python's BeautifulSoup and others are easy to use, easier than writing complex and brittle regular expressions.

perl -MHTML::TreeBuilder -le '
    $html = HTML::TreeBuilder->new_from_file($ARGV[0]) or die $!;
    foreach ($html->look_down(_tag => "tr")) {
        print map {$_->as_HTML()} $_->content_list();
    }
' input.html

or

python -c 'if True:
    import sys, BeautifulSoup
    html = BeautifulSoup.BeautifulSoup(open(sys.argv[1]).read())
    for tr in html.findAll("tr"):
        print "".join(tr.contents)
' input.html
share|improve this answer

sed and awk are not well suited for this task, you should rather use a proper html parser. For example hxselect from w3.org:

<htmlfile hxselect -s '\n' -c 'tr'
share|improve this answer
    
I dunno if hxselect is the best choice; I haven't used it but the man page does say it "reads a well-formed XML document" which many html documents are not. Probably worth a try tho. The html parser libs available for perl, python, et. al. will be much better, if that is an option. –  goldilocks Feb 13 '13 at 12:44
1  
@goldilocks: Best choice depends on the situation. In my experience hxselect does pretty good job with well-formed html/xml documents. Also, it's faster to use than perl, python and others. I think hxselect is a good middle-ground between sed/awk and parser libs. –  Thor Feb 13 '13 at 13:00
1  
If it works that's great! I was just adding a caveat for TechJack in case it didn't -- since I had also recommended using some kind of parser ;) The programming lib ones are of course more awkward but should deal with anything remotely passable as html. –  goldilocks Feb 13 '13 at 13:56
    
Thor, hxselect looks good, will definitely explore it more. Thanks. –  TechJack Feb 13 '13 at 17:16

If ruby is available you can do the following

ruby -e 'puts readlines.join[/(?<=<tr>).+(?=<\/tr>)/m].gsub(/<\/?tr>/, "")' file

where file is your input html file. The command executes a Ruby one-liner. First, it reads all lines from file and joins them to a string, readlines.join. Then, from the string it selects anything between (but not including) <tr> and <\/tr> that is one character or longer irrespective of newlines, [/(?<=<tr>).+(?=<\/tr>)/m]. Then, it removes any <tr> or </tr> from the string, gsub(/<\/?tr>/, "") (this is necessary to handle nested tr tags). Finally, it prints the string, puts.

You said that a html parser is not warranted for you but it is very easy to use Nokogiri with ruby and it makes the command simpler.

ruby -rnokogiri -e 'puts Nokogiri::HTML(readlines.join).xpath("//tr").map { |e| e.content }' file

-rnokogiri loads Nokogiri. Nokogiri::HTML(readlines.join) reads all lines of file. xpath("//tr") picks out every tr element and map { |e| e.content } picks out the content for each element, i.e. what is between <tr> and </tr>.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.