37

What would be the most straightforward way of making a GET request to a url over HTTPS, and getting the raw, unparsed response?

Could this be achieved with curl? If so, what options would you need to use?

49

If you want to use curl, this should work:

curl -D - https://www.google.com/

Note, however, that this is not exactly the raw response. For instance chunked transfer encoding will not be visible in the response. Using --raw solves this, also verbose mode (-v) is useful, too and -i shows the headers before the response body:

curl -iv --raw https://www.google.com/

If you want to use a pager like less on the result, it is also necessary to disable the progress-bar (-s):

curl -ivs --raw https://www.google.com/ | less

Depending on what you want to do this may or may not be a problem.

What you do get is all HTTP response headers and the document at the requested URL.

12

Here's a simple way that comes to mind

echo 'GET / HTTP/1.1
Host: google.com

' | openssl s_client -quiet -connect google.com:443 2>/dev/null
  • note that some servers (e.g github.com) only work with HTTP/1.0 – Steven Penny Jan 23 '18 at 1:14
4

It's not curl, but it should be available on almost all Unices:

wget -S --spider https://encrypted.site

If the status messages bother you:

wget -S --spider https://encrypted.site 2>&1 | awk '/^  /'

If you want CRLF line endings:

wget -S --spider https://encrypted.site 2>&1 | awk '/^  / { sub(/$/,"\r"); print }'
  • Do you know if this would preserve the \r\n characters in the response? – Acorn Jan 19 '12 at 0:04
  • @Acorn - First of all, not all webservers will respond with \r\n as a line ending. I don't believe this will preserve it in any case, but if that matters to you, I'll add a way of getting that result in the answer. – Chris Down Jan 19 '12 at 8:05
1
$ GET -e https://www.google.com

On Debian/Ubuntu distros belongs to the package lwp-request.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.