Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am extracting certain information from a website using curl command. I need to extract a certain subsection of the string using grep command. For example, using curl command I am getting the HTML page and storing in a variable as below.

page="$(curl http://www.somewebsite.com)"

My extracted HTML page is as below.

<html>
<body>
--------
--------
--------
<div class="section" id="id1" style="display:block">
<div class="header">
      <h2>Sub heading here
      </h2>
</div>
<div class="content">
<ol class="class1" id="id1">
<li class="">
<span class="jellybean">
</div>
----------
----------
</html>

Now, I need to extract the subsection of the entire div with id="id1". I tried the below command.

output="$(curl http:/somesite.com | grep "id="id1")"

It just gets me the 2 lines where I have id=id1. I need to extract the entire contents between the and tag.

share|improve this question
3  
Careful: you're going to make the unholy child weep the blood of virgins!. –  Joseph R. Feb 8 at 0:47
    
I found w3m command to scrap out HTML contents in a page. Not explored it completely. But seems interesting. Will update the answer if I found something useful. –  Ramesh Feb 8 at 1:00
    
@JosephR. the sample above is not valid XML. –  Stéphane Chazelas Feb 8 at 11:46

1 Answer 1

If the file were correct xml, you could use an xml parsing tool.

Otherwise, if there were no other (nested) div section inside that section, you could have done:

pcregrep -Mo '(?s)<div[^>]*id="id1".*?</div>' the-file.html

Here, you could try something like:

awk -vRS='<' '
  inside || /^div[^>]*id="id1"/ {
    inside = 1
    if (/^div/)
      n++
    else if (/^\/div>/ && !--n) {
      $0="/div>\n"
      inside=0
    }
    printf "<%s", $0
  }' the-file.html
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.