2

Given this:

<p>Currencies fluctuate every day. The rate shown is effective for transactions submitted to Visa on <strong>February 5, 2017</strong>, with a bank foreign transaction fee of <st <span><strong>1</strong> Euro = <strong>1.079992</strong> United States Dolla <p>The 'currency calculator' below gives you an indication of the cost of purchas <p>February 5, 2017</p><div class="clear-both"></div> <!-- removed clearboth- <p><strong>1 EUR = 1.079992 USD</strong></p> <div class="clear-both"></di <table width="290" border="0" cellspacing="0" cellpadding="3"> <a href="/content/VISA/US/en_us/home/support/consumer/travel-support/exchange e-calculator.html"> <button class="btn btn-default btn-xs"><span class="retur <p><p>This converter uses a single rate per day with respect to any two currencies. Rates displayed may not precisely reflect actual rate applied to transaction amount due to rounding differences, Rates apply to the date the transaction was processed by Visa; this may differ from the actual date of the transaction. Banks may or may not assess foreign transaction fees on cross-border transactions. Fees are applied at banks’ discretion. Please contact your bank for more information.</p>

I need to extract 1.079992

I'm using:

sed -E 's:.*(1\.[0-9\.]+).*:\1:g

... which works ... but is there a more elegant way?

Alternatively, is there a way to get that value straight from curl?

(My full command is: curl 'https://usa.visa.com/support/consumer/travel-support/exchange-rate-calculator.html/?fromCurr=USD&toCurr=EUR&fee=0&exchangedate=02/05/2017' | grep '<p><strong>1' | sed -E 's:.*(1\.[0-9\\.]+).*:\1:g' )

0

Using curl to fetch, lynx to parse, and awk to extract

Please don't parse XML/HTML with sed, grep, etc. HTML is context-free, but sed and friends are only regular.1

url='https://usa.visa.com/support/consumer/travel-support/exchange-rate-calculator.html/?fromCurr=USD&toCurr=EUR&fee=0&exchangedate=02/05/2017'
user_agent= 'Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0'

curl -sA "${user_agent}" "${url}"  \
| lynx -stdin -dump                \
| awk '/1 EUR/{ print $4 }'

You need some kind of HTML parser to reliably extract content. Here, I use lynx (a text-based web browser), but lighter alternatives exist.

Here, curl retrieves the page, then lynx parses it and dumps a textual representation. The /1 EUR/ causes awk to search for the string 1 EUR, finding only the line:

   1 EUR = 1.079992 USD

Then { print $4 } makes it print the fourth column, 1.079992.

Alternative solution without curl

Since my HTML parser of choice is lynx, curl is not necessary:

url='https://usa.visa.com/support/consumer/travel-support/exchange-rate-calculator.html/?fromCurr=USD&toCurr=EUR&fee=0&exchangedate=02/05/2017'
user_agent= 'Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0'

lynx -useragent="${user_agent}" -dump "${url}"  \
| awk '/1 EUR/{ print $4 }'

1 A pcre (grep -P in some implementations) can describe some context-free or even context-sensitive stringsets, but not all of them.


Edited 2017-12-23 to add a user-agent string (pretending to be Firefox), as the site currently blocks curl and lynx.

  • Rakesh's solution above works ... so what is the advantage of using lynx? Is it faster? – Dan Mar 7 '17 at 10:23
  • @Ze'ev The advantage of lynx is that it actually parses HTML correctly, so it's a more general solution. If they make substantial changes to that page, this will be easier to adapt – Fox Mar 7 '17 at 11:32
1

Another solution: html2text

curl -s 'https://usa.visa.com/support/consumer/travel-support/exchange-rate-calculator.html/?fromCurr=USD&toCurr=EUR&fee=0&exchangedate=2/12/2017' \
| html2text \
| grep '1 Euro' \
| awk '{ print $4 }'
0

Pipe the output of curl to the following grep command:

grep --color -Po '(?<=<strong>1 EUR = )\d+\.\d+(?= USD</strong>)'
  • That gives me curl: (23) Failed writing body (0 != 11805) -- maybe because I'm on a Mac without full curl? – Dan Mar 3 '17 at 11:59
  • First make sure that yor curl command is fully executing by itself before you pipe it to grep. – user218374 Mar 3 '17 at 12:00
0

Sugestion: use xml/html aware tools:

xmllint

curl "$url" | xmllint -html -xpath '//span/strong[2]/text()' - 

xidel

curl "$url" | xidel -s -e "//span/strong[2]" -

or even

xidel -e "/span/strong[2]" $url
0

I would use pandoc to convert to json, then python to extract the data. It will be much more robust than grep.

Like this, it takes input via stdin:

pandoc  -f html -t json | python3 -c '
import json
import sys

output=[]
data = json.load(sys.stdin)

for i in data[1][0]["c"]:
    if i["t"]=="Strong":
        output.append((i["c"]))

print(output[2][0]["c"])
'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.