1

I was trying to download data using Yahoo Finance with wget but ran into some issues with getting cookies and crumb. I used BTC-USD in this case and I saved the crumb data in a file named "crumbstore.txt".

wget --no-check-certificate --save-cookies=cookie.txt "https://finance.yahoo.com/quote/BTC-USD/history?p=BTC-USD" -O crumbstore.txt

From there I wanted to extract the crumb string and insert it into the quote link shown below ($crumb). The time period is from 2017-01-01 to 2020-01-06 in this case.

wget --no-check-certificate --load-cookies=cookie.txt "https://query1.finance.yahoo.com/v7/finance/download/BTC-USD?period1=1483265281&period2=1578305313&interval=1d&events=history&crumb=$crumb" -O BTC-USD_price.csv

In order to see how the "crumbstore.txt" file is structured, I ran the following code with grep.

grep -o ".\{0,50\}CrumbStore.\{0,50\}" crumbstore.txt

I was able to see what I needed to get was the series of letters and numbers stored in "CrumbStore":{"crumb":"XXXXXXXXXXX"}.

Here is a snippet of crumbstore.txt and the essential task is to extract "BdpohsJbyt." from the text.

"secondary_accounts":{"list":"Pos(r) M(0) P(0) List(n)","list_item":"Bdt(asMenuBorder)","anchor":"Td(n) Td(n):h Py(16px) D(b) Cf C(#000) Bgc(asMenuHoverBgc):h"}}}}},"CrumbStore":{"crumb":"BdpohsJbyt."},"StreamStore":{"articleCategory":{"term":"","label":""}
  • you can use python to get information from web page. BeautifulSoup is good library for your job. it will be better for you. – durmus yılmaz Jan 6 at 11:06
1

Your data seems to be JSON formatted, and unless you work in a very restrictive environment where dirty hacks are preferred over installing the right tools, you should just install jq (available in most distro repositories) and use that.

Without seeing the actual data, it's not possible to write down the correct jq invocation.

| improve this answer | |
0

How about deleting everything before and after the string, like

sed -n 's/^.*crumb":"//; s/".*$//p' crumb.store
XXXXXXXXXXX
| improve this answer | |
0

it is not so clear for me what do you want to get or change. I grabbed the file on the way you did it, and extracted the string you may need from the crumbstore.txt file:

cat crumbstore.txt | grep -oE 'cru.\w*.:.{13}' | grep -Ev '{|pf'

which gave this result:

    crumb":"q7md5ctTst0"
    crumb":"WvBqKN8S0E5"
    crumb":"q7md5ctTst0"
    crumb":"q7md5ctTst0"
    crumb":"q7md5ctTst0"

And to extract the 11-char string from quotes I used awk

cat crumbstore.txt | grep -oE 'cru.\w*.:.{13}' | grep -Ev '{|pf' | awk -F'"' '{print $3}'

with this result:

    q7md5ctTst0
    WvBqKN8S0E5
    q7md5ctTst0
    q7md5ctTst0
    q7md5ctTst0

Is this what you whant?

| improve this answer | |
  • Yes this looks awesome. Thank you! – Jason Li Jan 10 at 9:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.