Need to extract a number from HTML

Question

Given this:

<p>Currencies fluctuate every day. The rate shown is effective for transactions submitted to Visa on <strong>February 5, 2017</strong>, with a bank foreign transaction fee of <st <span><strong>1</strong> Euro = <strong>1.079992</strong> United States Dolla <p>The 'currency calculator' below gives you an indication of the cost of purchas <p>February 5, 2017</p><div class="clear-both"></div> <!-- removed clearboth- <p><strong>1 EUR = 1.079992 USD</strong></p> <div class="clear-both"></di <table width="290" border="0" cellspacing="0" cellpadding="3"> <a href="/content/VISA/US/en_us/home/support/consumer/travel-support/exchange e-calculator.html"> <button class="btn btn-default btn-xs"><span class="retur <p><p>This converter uses a single rate per day with respect to any two currencies. Rates displayed may not precisely reflect actual rate applied to transaction amount due to rounding differences, Rates apply to the date the transaction was processed by Visa; this may differ from the actual date of the transaction. Banks may or may not assess foreign transaction fees on cross-border transactions. Fees are applied at banks’ discretion. Please contact your bank for more information.</p>

I need to extract 1.079992

I'm using:

sed -E 's:.*(1\.[0-9\.]+).*:\1:g

... which works ... but is there a more elegant way?

Alternatively, is there a way to get that value straight from curl?

(My full command is: curl 'https://usa.visa.com/support/consumer/travel-support/exchange-rate-calculator.html/?fromCurr=USD&toCurr=EUR&fee=0&exchangedate=02/05/2017' | grep '<p><strong>1' | sed -E 's:.*(1\.[0-9\\.]+).*:\1:g' )

Answer 1

Using curl to fetch, lynx to parse, and awk to extract

Please don't parse XML/HTML with sed, grep, etc. HTML is context-free, but sed and friends are only regular.¹

url='https://usa.visa.com/support/consumer/travel-support/exchange-rate-calculator.html/?fromCurr=USD&toCurr=EUR&fee=0&exchangedate=02/05/2017'
user_agent= 'Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0'

curl -sA "${user_agent}" "${url}"  \
| lynx -stdin -dump                \
| awk '/1 EUR/{ print $4 }'

You need some kind of HTML parser to reliably extract content. Here, I use lynx (a text-based web browser), but lighter alternatives exist.

Here, curl retrieves the page, then lynx parses it and dumps a textual representation. The /1 EUR/ causes awk to search for the string 1 EUR, finding only the line:

   1 EUR = 1.079992 USD

Then { print $4 } makes it print the fourth column, 1.079992.

Alternative solution without curl

Since my HTML parser of choice is lynx, curl is not necessary:

url='https://usa.visa.com/support/consumer/travel-support/exchange-rate-calculator.html/?fromCurr=USD&toCurr=EUR&fee=0&exchangedate=02/05/2017'
user_agent= 'Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0'

lynx -useragent="${user_agent}" -dump "${url}"  \
| awk '/1 EUR/{ print $4 }'

---

¹ A pcre (grep -P in some implementations) can describe some context-free or even context-sensitive stringsets, but not all of them.

---

Edited 2017-12-23 to add a user-agent string (pretending to be Firefox), as the site currently blocks curl and lynx.

Answer 2

Another solution: html2text

curl -s 'https://usa.visa.com/support/consumer/travel-support/exchange-rate-calculator.html/?fromCurr=USD&toCurr=EUR&fee=0&exchangedate=2/12/2017' \
| html2text \
| grep '1 Euro' \
| awk '{ print $4 }'

Answer 3

Sugestion: use xml/html aware tools:

xmllint

curl "$url" | xmllint -html -xpath '//span/strong[2]/text()' - 

xidel

curl "$url" | xidel -s -e "//span/strong[2]" -

or even

xidel -e "/span/strong[2]" $url

Answer 4

Pipe the output of curl to the following grep command:

grep --color -Po '(?<=<strong>1 EUR = )\d+\.\d+(?= USD</strong>)'

Answer 5

I would use pandoc to convert to json, then python to extract the data. It will be much more robust than grep.

Like this, it takes input via stdin:

pandoc  -f html -t json | python3 -c '
import json
import sys

output=[]
data = json.load(sys.stdin)

for i in data[1][0]["c"]:
    if i["t"]=="Strong":
        output.append((i["c"]))

print(output[2][0]["c"])
'