I am extracting certain information from a website using curl command. I need to extract a certain subsection of the string using grep command. For example, using curl command I am getting the HTML page and storing in a variable as below.

page="$(curl http://www.somewebsite.com)"

My extracted HTML page is as below.

<div class="section" id="id1" style="display:block">
<div class="header">
      <h2>Sub heading here
<div class="content">
<ol class="class1" id="id1">
<li class="">
<span class="jellybean">

Now, I need to extract the subsection of the entire div with id="id1". I tried the below command.

output="$(curl http:/somesite.com | grep "id="id1")"

It just gets me the 2 lines where I have id=id1. I need to extract the entire contents between the and tag.

  • 3
    Careful: you're going to make the unholy child weep the blood of virgins!.
    – Joseph R.
    Feb 8, 2014 at 0:47
  • I found w3m command to scrap out HTML contents in a page. Not explored it completely. But seems interesting. Will update the answer if I found something useful.
    – Ramesh
    Feb 8, 2014 at 1:00
  • @JosephR. the sample above is not valid XML. Feb 8, 2014 at 11:46

1 Answer 1


If the file were correct xml, you could use an xml parsing tool.

Otherwise, if there were no other (nested) div section inside that section, you could have done:

pcregrep -Mo '(?s)<div[^>]*id="id1".*?</div>' the-file.html

Here, you could try something like:

awk -vRS='<' '
  inside || /^div[^>]*id="id1"/ {
    inside = 1
    if (/^div/)
    else if (/^\/div>/ && !--n) {
    printf "<%s", $0
  }' the-file.html

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.