Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a couple of hundred html source code files. I need to extract the contents of a particular <div> element from each of these file so I'm going to write a script to loop through each file. The element structure is like this:

<div id='the_div_id'>
  <div id='some_other_div'>
  <h3>Some content</h3>
  </div>
</div>

Can anyone suggest a method by which I can extract the div the_div_id and all the child elements and content from a file using the linux command line?

share|improve this question
add comment

2 Answers

up vote 15 down vote accepted

The html-xml-utils package, available in most major Linux distributions, has a number of tools that are useful when dealing with HTML and XML documents. Particularly useful for your case is hxselect which reads from standard input and extracts elements based on CSS selectors. Your use case would look like:

hxselect '#the_div_id' <file

You might get a complaint about input not being well formed depending on what you are feeding it. This complaint is given over standard error and thus can be easily suppressed if needed. An alternative to this would to be to use Perl's HTML::PARSER package; however, I will leave that to someone with Perl skills less rusty than my own.

share|improve this answer
    
Brilliant, thanks! –  conorgriffin Jan 24 '11 at 20:22
add comment

Here's an untested Perl script that extracts <div id="the_div_id"> elements and their contents using HTML::TreeBuilder.

#!/usr/bin/env perl
use strict;
use warnings
use HTML::TreeBuilder;
foreach my $file_name (@ARGV) {
    my $tree = HTML::TreeBuilder->new;
    $tree->parse_file($file_name);
    for my $subtree ($tree->look_down(_tag => "div", id => "the_div_id")) {
        my $html = $subtree->as_HTML;
        $html =~ s/(?<!\n)\z/\n/;
        print $html;
    }
    $tree = $tree->delete;
}

If you're allergic to Perl, Python has HTMLParser.

P.S. Do not try using regular expressions..

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.