Need to extract a number from HTML

Given this:

`<p>Currencies fluctuate every day. The rate shown is effective for transactions submitted to Visa on <strong>February 5, 2017</strong>, with a bank foreign transaction fee of <st <span><strong>1</strong> Euro = <strong>1.079992</strong> United States Dolla <p>The 'currency calculator' below gives you an indication of the cost of purchas <p>February 5, 2017</p><div class="clear-both"></div> <!-- removed clearboth- <p><strong>1 EUR = 1.079992 USD</strong></p> <div class="clear-both"></di <table width="290" border="0" cellspacing="0" cellpadding="3"> <a href="/content/VISA/US/en_us/home/support/consumer/travel-support/exchange e-calculator.html"> <button class="btn btn-default btn-xs"><span class="retur <p><p>This converter uses a single rate per day with respect to any two currencies. Rates displayed may not precisely reflect actual rate applied to transaction amount due to rounding differences, Rates apply to the date the transaction was processed by Visa; this may differ from the actual date of the transaction. Banks may or may not assess foreign transaction fees on cross-border transactions. Fees are applied at banks’ discretion. Please contact your bank for more information.</p>`

I need to extract `1.079992`

I'm using:

``````sed -E 's:.*(1\.[0-9\.]+).*:\1:g
``````

... which works ... but is there a more elegant way?

Alternatively, is there a way to get that value straight from `curl`?

(My full command is: `curl 'https://usa.visa.com/support/consumer/travel-support/exchange-rate-calculator.html/?fromCurr=USD&toCurr=EUR&fee=0&exchangedate=02/05/2017' | grep '<p><strong>1' | sed -E 's:.*(1\.[0-9\\.]+).*:\1:g'` )

Using `curl` to fetch, `lynx` to parse, and `awk` to extract

Please don't parse XML/HTML with `sed`, `grep`, etc. HTML is context-free, but `sed` and friends are only regular.1

``````url='https://usa.visa.com/support/consumer/travel-support/exchange-rate-calculator.html/?fromCurr=USD&toCurr=EUR&fee=0&exchangedate=02/05/2017'
user_agent= 'Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0'

curl -sA "\${user_agent}" "\${url}"  \
| lynx -stdin -dump                \
| awk '/1 EUR/{ print \$4 }'
``````

You need some kind of HTML parser to reliably extract content. Here, I use `lynx` (a text-based web browser), but lighter alternatives exist.

Here, `curl` retrieves the page, then `lynx` parses it and dumps a textual representation. The `/1 EUR/` causes `awk` to search for the string `1 EUR`, finding only the line:

``````   1 EUR = 1.079992 USD
``````

Then `{ print \$4 }` makes it print the fourth column, `1.079992`.

Alternative solution without `curl`

Since my HTML parser of choice is `lynx`, `curl` is not necessary:

``````url='https://usa.visa.com/support/consumer/travel-support/exchange-rate-calculator.html/?fromCurr=USD&toCurr=EUR&fee=0&exchangedate=02/05/2017'
user_agent= 'Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0'

lynx -useragent="\${user_agent}" -dump "\${url}"  \
| awk '/1 EUR/{ print \$4 }'
``````

1 A `pcre` (`grep -P` in some implementations) can describe some context-free or even context-sensitive stringsets, but not all of them.

Edited 2017-12-23 to add a user-agent string (pretending to be Firefox), as the site currently blocks `curl` and `lynx`.

• Rakesh's solution above works ... so what is the advantage of using lynx? Is it faster? – Dan Mar 7 '17 at 10:23
• @Ze'ev The advantage of `lynx` is that it actually parses HTML correctly, so it's a more general solution. If they make substantial changes to that page, this will be easier to adapt – Fox Mar 7 '17 at 11:32

Another solution: `html2text`

``````curl -s 'https://usa.visa.com/support/consumer/travel-support/exchange-rate-calculator.html/?fromCurr=USD&toCurr=EUR&fee=0&exchangedate=2/12/2017' \
| html2text \
| grep '1 Euro' \
| awk '{ print \$4 }'
``````

Pipe the output of `curl` to the following `grep` command:

``````grep --color -Po '(?<=<strong>1 EUR = )\d+\.\d+(?= USD</strong>)'
``````
• That gives me `curl: (23) Failed writing body (0 != 11805)` -- maybe because I'm on a Mac without full `curl`? – Dan Mar 3 '17 at 11:59
• First make sure that yor `curl` command is fully executing by itself before you pipe it to `grep`. – user218374 Mar 3 '17 at 12:00

Sugestion: use xml/html aware tools:

xmllint

``````curl "\$url" | xmllint -html -xpath '//span/strong[2]/text()' -
``````

xidel

``````curl "\$url" | xidel -s -e "//span/strong[2]" -
``````

or even

``````xidel -e "/span/strong[2]" \$url
``````

I would use `pandoc` to convert to `json`, then `python` to extract the data. It will be much more robust than `grep`.

Like this, it takes input via stdin:

``````pandoc  -f html -t json | python3 -c '
import json
import sys

output=[]