0

I am wondering why certain URLs load without a hassle in a browser tab but timeout with tools like wget and curl.
Here is an example URL:
https://www.nasdaq.com/market-activity/stocks/gme/news-headlines

I have tried creating download scripts for this webpage using curl and wget but without success (i.e. they all hang up).

wget --adjust-extension -t 2 -U 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36' "https://www.nasdaq.com/market-activity/stocks/gme/news-headlines" -O gme.html

wget --adjust-extension -t 2 -U 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36' "www.nasdaq.com/market-activity/stocks/gme/news-headlines" -O gme.html

curl -o 01.html https://www.nasdaq.com/market-activity/stocks/gme/news-headlines

curl -o 01.html www.nasdaq.com/market-activity/stocks/gme/news-headlines
2
  • This is totally offtopic, but the right strategy is to open the network tools pane in your browser (e.g. with Ctrl-Shift-E in firefox) and from there right click on the request then Copy => Copy As cURL which will give you a curl command line which you could then adjust. In your case, you could see that that site works without a Cookie header, but NOT without the right Referer and User-Agent. It would also need keep-alive and only send compressed data.
    – user313992
    Commented Jan 31, 2021 at 23:30
  • @user414777 If this is off topic, I'd expect to see a close vote from you.
    – Kusalananda
    Commented Feb 1, 2021 at 8:19

1 Answer 1

0

Their web server may well look at the User-Agent: header in the request and refuse to server requests coming from curl or wget and similar tools. They may choose to do this because they might not want scripted access to their web content. In fact, part of their terms of service (which you agree to by merely accessing their site) says

Use of the Services. To use our Services you must:

[...]

Not use any automated process to access or use the Service or any process, whether automated or manual, to capture data or content from the Service or circumvent any mechanisms for preventing the unauthorized reproduction or distribution of the Service for any reason;

Changing the User-Agent: header used in the request is a matter of finding the correct options in the manual of your tool, deciding whether using that option would be legal or not in your use case, and acting on that consideration.

1
  • 1
    The web server looks at each and every header (not just the User-Agent) and this is not the right place for legal advice.
    – user313992
    Commented Jan 31, 2021 at 23:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .