This is driving me insane, as I'm sure there must be some program doing the job but I just can't find it.
To be noted, I'd be on ARM/Raspberry so unfortunately Xidel is not an option here, unless someone is kind enough to point me in the right direction for compiling it myself (if possible at all). I also tried w3m but I'm not quite sure dumping the page to another to-be-formatted file would truly help, but then again please prove me wrong!

Basically I need to read an https url, then parse the html to something easily accessible from within Bash, and finally store some values in a db.
I'm already doing most of the job simply with grep, but as soon as the data I need is not exactly how it should be formatted I'm stuck. See example:

<h1>Title</h1> #easy to extract Title only with grep | sed

</h1> #no way

So you can see the catch here, what I need is a script solution that reads the whole page and gives me an array of the contents, rather than the 'randomly' formatted output I currently have.
Thanks everyone


I would filter the file using tidy, to put each top-level tag on a separate line, and use something better suited to this type of scripting (such as Perl) for handling the markup. sed is most useful when the changes are very simple.

For example, a fragment from this page (after filtering) would look like this (using tidy -wrap 4096 to reduce wrapped lines):

<body class="question-page new-topbar">
<div id="noscript-padding"></div>
<div id="notify-container"></div>
<div id="overlay-header"></div>
<div id="custom-header"></div>
<div class="topbar">
<div class="topbar-wrapper">
<div class="js-topbar-dialog-corral">
<div class="topbar-dialog siteSwitcher-dialog dno">
<div class="header">
<h3><a href="//unix.stackexchange.com">current community</a></h3>
<div class="modal-content current-site-container">
<ul class="current-site">

The "a" tags, etc., will be inline, but the page structure is simpler to work with.

I "downloaded" the page using lynx -source. You could use a variety of tools however: wget and curl are more often used for this purpose.

| improve this answer | |

Have a try at pup. i.e. for OSX:

brew install https://raw.githubusercontent.com/EricChiang/pup/master/pup.rb

You can use CSS3 selector and extractor to get value from href attributes.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.