Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am a beginner linux user, trying to teach myself how to use linux tools on a cygwin install. I decided to make up a project to try to teach myself the basics of shell scripting and simultaneously accomplish some personal education. My original project was to save the html pages from every winner of the Sakharov Prize into a folder, and to write a script that would process all the html text files and return the name, year, birth and death in a hyphenated format, and country of origin. Due to certain inconsistencies in date formatting (18 July 1918 vs January 23, 1938), as well as an inability to handle dead people vs live people with no death date or figure out how to tell a computer how to recognize country names without manually listing out all the countries myself, I've basically given up on this project.

Now, I'm just trying to return the year, name, and country of origin of each recipient from the html table taken from the Sakharov Prize wikipedia page.

So, given the following sample html:

<tr>
<td>1988</td>
<td><span style="display:none;">Mandela, Nelson</span><span class="vcard"><span class="fn"><a href="/wiki/Nelson_Mandela" title="Nelson Mandela">Nelson Mandela</a></span></span></td>
<td><a href="/wiki/South_Africa" title="South Africa">South Africa</a></td>
<td>Anti-apartheid activist and later President of South Africa</td>
<td><sup id="cite_ref-twentyyears_5-0" class="reference"><a href="#cite_note-twentyyears-5"><span>[</span>5<span>]</span></a></sup></td>
</tr>
<tr>
<td>1988</td>
<td><span style="display:none;">Marchenko, Anatoly</span><span class="vcard"><span class="fn"><a href="/wiki/Anatoly_Marchenko" title="Anatoly Marchenko">Anatoly Marchenko</a></span></span> (posthumously)</td>
<td><a href="/wiki/Soviet_Union" title="Soviet Union">Soviet Union</a></td>
<td>Soviet dissident, author and humans rights activist</td>
<td><sup id="cite_ref-twentyyears_5-1" class="reference"><a href="#cite_note-twentyyears-5"><span>[</span>5<span>]</span></a></sup></td>
</tr>

what is the best way to return just the year, name, and country of origin of each recipient? Right now I'm thinking about just writing an awk script that returns everything that does not match /<*>/, but that is not exactly what I want. Can someone give me some pointers or ideas of how to pick out the names, year, and countries specifically? Or at least some books with better and more manageable sample problems than ones that I could come up with myself? None of this sounded unreasonable when I started...

share|improve this question
9  
3  
Ditto. Which is not to say that it can't be done in this particular case or any particular case, but in general mark-up languages should be parsed with a parser designed for the task. This is because a method not based on decomposing the structure formally will become nothing but a headache as you try to apply it to more and more generalized cases. Using a particular case as an exercise will not teach you good habits. Crass analogy: managing to fix a car with dinner utensils is not a worthwhile exercise, even if you did it "this time" [quite the SO case study in jordanm's link btw...] –  goldilocks Mar 20 '13 at 5:52
1  
Agreed, and on top of that I find cygwin to be more of a hard-core enthusiast type of environment than a productive environment. Particularly as a self-professed newbie you now need to wrestle the peculiarities and eccentricities (Is that a word?) of Cygwin while at the same time trying to learn something about the tools. I highly suggest a full Linux installation in a VM like VirtualBox, or even a second PC under the desk to which you can connect. –  Johan Mar 20 '13 at 8:11
    
Appreciate the insight, guys. Lesson learned. –  tzisc Mar 20 '13 at 17:35
add comment

3 Answers 3

up vote 3 down vote accepted

As has been mentioned regex is not good for parsing html. Similar to another parse answer you can make a Ruby one-liner such as the following to do it for you. Note that it requires Nokogiri which you can install as a gem (sudo gem install nokogiri).

ruby -rnokogiri -e 'Nokogiri::HTML(readlines.join).css("tr").each { |tr| tr.xpath(".//td").take(3).each { |td| puts td.content } }' sample.html

It reads the given file, in this case sample.html, gets all tr elements and for each such element it prints the content of the first three td elements.

For your sample it will output:

1988
Mandela, NelsonNelson Mandela
South Africa
1988
Marchenko, AnatolyAnatoly Marchenko (posthumously)
Soviet Union

The problem is the lines which contains the names twice, e.g. (formatted to be easier to read)

<td>
  <span style="display:none;">Mandela, Nelson</span>
  <span class="vcard"><span class="fn">
      <a href="/wiki/Nelson_Mandela" title="Nelson Mandela">Nelson Mandela</a>
    </span>
  </span>
</td>

in which the name is first in a span with style="display:none;" and then in again in another span. I am not sure how to extract only the name that is not within an element with style="display:none;. (I have found http://stackoverflow.com/q/6096327/789593 and http://stackoverflow.com/q/11602077/789593 but they do not describe the right tecnique. Perhaps someone can come up with a fix via http://nokogiri.org/Nokogiri/XML/Node.html?)

share|improve this answer
    
Appreciate it. I'll grab the ruby package and give it a shot today and let you know how it goes. –  tzisc Mar 20 '13 at 17:37
    
@tzisc If you are on a Debian based system sudo apt-get install ruby1.9.1-dev should be enough install the ruby stuff you need then you can grab nokogiri as I described in the answer. –  N.N. Mar 20 '13 at 20:15
    
So I got the ruby package on my ubuntu vps and messed around for a bit with nokogiri. I think my lack of understanding of ruby syntax is holding me back a bit, as I don't really understand the tutorials that they have on the site, but you gave me the insight of trying to use a jquery script to try and do this. I think I'm on the right track now, so thank you. –  tzisc Mar 21 '13 at 20:43
    
@tzisc Ruby is not hard to learn. Maybe you just lack the understanding of its iterators, see e.g. www-rohan.sdsu.edu/doc/ruby/chp_03/iterators.html. Otherwise, see ruby-doc.org/docs/ProgrammingRuby for learning Ruby as a whole. –  N.N. Mar 22 '13 at 5:51
add comment

For HTML pages, I often use the w3m -dump command first to dump the formatted page to stdout and do more processing with other tools like grep, sed, awk, ... You can give it a try.

share|improve this answer
add comment

I have created node.js package which can be used here: gumba. It's kind of awk, sed replacement.

so in your example it will work like this:

cat file.html | gumba "stripTags()"

which outputs:

1988
Mandela, NelsonNelson Mandela
South Africa
Anti-apartheid activist and later President of South Africa
[5]


1988
Marchenko, AnatolyAnatoly Marchenko (posthumously)
Soviet Union
Soviet dissident, author and humans rights activist
[5]

Although here I think it's better not to use oneliners, but to actually write script in any language you know.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.