3

I tried to scrape Google image search result page with curl from Terminal, but it doesn't give me an actual html source that I can view with "View Page Source" in Firefox. I tried both "curl [url]" and "curl -L [url]". Both gave me a short html source that includes "Your client does not have permission to get URL " "from this server". How can I get the html source that I can get in Firefox with a shell script?

Part of the short html I got in Terminal said this.

Please see Google's Terms of Service posted at http://www.google.com/terms_of_service.html

If you believe that you have received this response in error, please report your problem. However, please make sure to take a look at our Terms of Service (http://www.google.com/terms_of_service.html). In your email, please send us the entire code displayed below.

  • @JigglyNaga yes. I edited my question. – stacko Aug 23 '16 at 12:36
  • Just checking, were you polite and waited a couple of seconds between each request? – grochmal Aug 23 '16 at 13:11
2

The error message contains a broken link, but Google's current terms of service say:

Do not misuse our Services, for example, do not interfere with our Services or try to access them using a method other than the interface and the instructions that we provide.

(emphasis mine)

They're refusing your request for some reason. It could be that they've seen suspicious activity from your IP address, but it's most probably that they've spotted that you're using curl instead of a regular browser (in which you would see the adverts).

You could make curl imitate such a browser, by providing a common user-agent (eg. from http://www.browser-info.net/useragents) to the -A option, but that would still be violating the ToS.

2

I would think it's most likely an issue with cookies (or lack thereof). You may want to try playing around with curl's -c [file] option for its cookie jar, saving cookies to a file. You may have to script a curl request to say "google.com" to submit the image search and save any cookies to file, and then a second query to scrape the web url you're given for the actual search and load the saved cookies using the -b [file] option. It may be that it just needs the session ID cookie.

You can see in the screenshot below that Google assigns a number of cookie variables.

Cookies example for google images

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.