Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I need extract text with a bash script from a website in HTML, I used this solution but doesn't work fine for me, because I must edit and format the output text. I need the text between the tag:

<p><p tabindex="0">

Example I am surf in https://apps.ubuntu.com/cat/applications/clementine/ In Firefox displayed the next tag in your source code:

<p><p tabindex="0">Clementine is a multiplatform music player focusing on a fast and easy-to-use interface for searching and playing your music.</p><p tabindex="0">Summary of included features :</p><ul><li>Search and play your local music library.</li><li>Listen to internet radio from Last.fm, SomaFM and Magnatune.</li><li>Tabbed playlists, import and export M3U, XSPF, PLS and ASX.</li><li>Visualisations from projectM.</li><li>Transcode music into MP3, Ogg Vorbis, Ogg Speex, FLAC or AAC</li><li>Edit tags on MP3 and OGG files, organise your music.</li><li>Download missing album cover art from Last.fm.</li><li>Native desktop notifications using libnotify.</li><li>Supports MPRIS, or remote control using the command-line.</li><li>Remote control using a Wii Remote, MPRIS or the command-line.</li><li>Copy music to your iPod, iPhone, MTP or mass-storage USB player.</li><li>Queue manager. It is largely a port of Amarok 1.4, with some features rewritten to take advantage of Qt4.</li></ul></p>

How to extract the above text, after <p><p tabindex="0"> ? remember is only a example, the website and tag could change.

Output='Clementine is a multiplatform music player focusing on a fast and easy-to-use interface for searching and playing your music.'


thanks to $John1024 your solution work fine with ubuntu site, but with opensuse doesn't display correctly. this is the variable with OpenSuse site:

output="$(wget -q http://software.opensuse.org/package/clementine -O - | sed -n 's/.*<p id="pkg-desc">\([^<]*\).*/\1/p')".

echo $output

Clementine is a modern music player and library organiser. Clementine is a (doesn't display completaly the text)

I tried also:

output="$(wget -q http://software.opensuse.org/package/clementine/ -O - | sed -ne '/<p id="pkg-desc">/, /[.</p>]$/p')"

echo $output

<p id="pkg-desc">Clementine is a modern music player and library organiser. Clementine is a port of Amarok 1.4, with some features rewritten to take advantage of Qt4. (it displays with '<p id="pkg-desc">')

share|improve this question
    
do you only need the first occurance? Or do you need all occurances of text between those tags? how should they be diferenciated? –  Eric Mar 16 at 3:52
    
So how do we know when to stop? Do you want the entire line? –  terdon Mar 16 at 4:36
    
thanks @terdon only the first information from <p><p tabindex="0"> to </p><p –  davidva Mar 16 at 7:00
add comment

1 Answer 1

up vote 2 down vote accepted

In general, one should use a tool that understands html. For limited purposes, though, a simple command may suffice. In this case, sed is sufficient to do what you ask and works well in bash scripts. If you have captured the source html into index.html, then:

$ sed -n 's/.*<p><p tabindex="0">\([^<]*\).*/\1/p' index.html 
Clementine is a multiplatform music player focusing on a fast and easy-to-use interface for searching and playing your music.

Or, to capture the html and process it all in one step:

$ wget -q https://apps.ubuntu.com/cat/applications/clementine/ -O - | sed -n 's/.*<p><p tabindex="0">\([^<]*\).*/\1/p' 
Clementine is a multiplatform music player focusing on a fast and easy-to-use interface for searching and playing your music.

To capture that output to a bash variable:

output="$(wget -q https://apps.ubuntu.com/cat/applications/clementine/ -O - | sed -n 's/.*<p><p tabindex="0">\([^<]*\).*/\1/p')"

The -n option is used on sed. This tells it not to print output unless we explicitly ask. sed goes through the input line by line looking for a line which matches .*<p><p tabindex="0">\([^<]*\).*. All the text that follows the <p tabindex="0"> and the next tag is captured in variable 1. Everything on that line is then replaced with just that captured text which is then printed.

share|improve this answer
    
Thanks!! work great!, thanks for explain me. –  davidva Mar 16 at 6:52
    
Hi, work fine with ubuntu site, but with opensuse doesn't display correctly. this is the variable with OpenSuse site: output="$(wget -q software.opensuse.org/package/clementine -O - | sed -n 's/.*<p id="pkg-desc">([^<]*).*/\1/p')". I tried also: output="$(wget -q software.opensuse.org/package/clementine -O - | sed -ne '/<p id="pkg-desc">/, /[.</p>]$/p')" –  davidva Mar 16 at 21:22
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.