2

I'm trying to use cURL to automate some processes that we usually do using a website.

I was able to login to the website using curl and the following command:

curl -k -v -i --user "[user]:[password]" -D cookiejar.txt https://link/to/home/page

However, when I'm trying to use the generated cookiejar.txt file for subsequent calls, I'm not getting passed the authorization.

The browser sends the following data to the server:

GET /[my other page] HTTP/1.1
Host    [my host]
User-Agent  Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Firefox/31.0
Accept  text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language en-US,en;q=0.5
Accept-Encoding gzip, deflate
Cookie  JSESSIONID=[my session id]
Authorization   Basic [my encrypted string]
Connection  keep-alive

So, I changed my second cURL call to something like this, to be sure that all these parameters are sent as well:

curl -i -X GET -k -v \
-b cookiejar.txt \
-H "Authorization: Basic [my encrypted string]" \
-H "Host: [my host]"  \
-H "User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Firefox/31.0" \
-H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" \
-H "Content-Type: application/x-www-form-urlencoded" \
-H "Accept-Language: en-US,en;q=0.5" \
-H "Connection: Keep-Alive" \
https://[my other page]

Unfortunately this doesn't work. If I omit the Authorization header, I get a 401 error. If I include it in my cURL request, I get the Login page (with the 200 OK response).

There's no error in the console to give me at least a hint about what the problem is.

I appreciate any idea to help me get passed this issue.

7
  • Are you sure no Referer is needed? Have you tried with wget? eventhough wget is old I found bugs in cURL that doesn't allow me to access some https pages, but with wget I managed to get what I wanted.
    – YoMismo
    Commented Feb 6, 2015 at 7:57
  • I added a Referer as well. I just forgot to add it here. I will try with wget too, thank you for the suggestion.
    – Ionut
    Commented Feb 6, 2015 at 8:02
  • I just tried with wget too. It behaves the same as cURL. :(
    – Ionut
    Commented Feb 6, 2015 at 8:21
  • Have you tried debuging curl? Does it send the same headers as the ones the web browser is sending?
    – YoMismo
    Commented Feb 6, 2015 at 8:48
  • Yes, they are the same, except the JSESSIONID cookie and the Accept-Encoding header (gzip, deflate). The cookie is logical to be different. The encoding header cannot be used otherwise the server response gets unintelligible.
    – Ionut
    Commented Feb 6, 2015 at 9:27

2 Answers 2

0

It could be due to a redirect during authorization. See the -L and --location-trusted options in man curl. Also try testing -w redirect_url to see the actual page you would be redirected to if that is the case.

   -L, --location
          (HTTP/HTTPS) If the server reports that the requested page has  moved  to  a  different  location
          (indicated  with a Location: header and a 3XX response code), this option will make curl redo the
          request on the new place. If used together with -i, --include or -I,  --head,  headers  from  all
          requested  pages  will  be shown. When authentication is used, curl only sends its credentials to
          the initial host. If a redirect takes curl to a different host, it won't be able to intercept the
          user+password.  See  also  --location-trusted  on how to change this. You can limit the amount of
          redirects to follow by using the --max-redirs option.

          When curl follows a redirect and the request is not a plain GET (for example  POST  or  PUT),  it
          will  do  the  following  request  with  a  GET if the HTTP response was 301, 302, or 303. If the
          response code was any other 3xx code, curl will re-send the  following  request  using  the  same
          unmodified method.

   --location-trusted
          (HTTP/HTTPS)  Like  -L,  --location, but will allow sending the name + password to all hosts that
          the site may redirect to. This may or may not introduce a security breach if the  site  redirects
          you  to  a  site to which you'll send your authentication info (which is plaintext in the case of
          HTTP Basic authentication).

   -w, --write-out <format>
          Defines what to display on stdout after a completed and successful operation.  The  format  is  a
          string  that  may contain plain text mixed with any number of variables. The string can be speci‐
          fied as "string", to get read from a particular file you specify it "@filename" and to tell  curl
          to read the format from stdin you write "@-".

          The  variables  present  in  the output format will be substituted by the value or text that curl
          thinks fit, as described below. All variables are specified as %{variable_name} and to  output  a
          normal  % you just write them as %%. You can output a newline by using \n, a carriage return with
          \r and a tab space with \t.

          NOTE: The %-symbol is a special symbol in the win32-environment, where all occurrences of %  must
          be doubled when using this option.

          The variables available are:


          redirect_url   When  an  HTTP request was made without -L to follow redirects, this variable will
                         show the actual URL a redirect would take you to. (Added in 7.18.2)
2
  • Thank you for the answer. I added the -L and --location-trusted to my query but in vain. Nothing changed. I also used the -w option with this parameter: "Response code: %{http_code}\n Redirect URL: %{redirect_url}\n URL Effective: %{url_effective}". Unfortunately, nothing special got out of it. The results were: "Response code: 200" and the redirect URL was empty. The url_effective was exactly the url that I passed to curl.
    – Ionut
    Commented Feb 6, 2015 at 9:28
  • Oh, well that rules that out then :)
    – iyrin
    Commented Feb 6, 2015 at 9:33
0

I was able to reach the desired page in the end.

It seems that I wasn't following the correct sequence of URL calls. Once I did that, the desired page was retrieved correctly.

Thank you very much for the quick responses !

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .