I'm trying to use cURL to automate some processes that we usually do using a website.

I was able to login to the website using curl and the following command:

curl -k -v -i --user "[user]:[password]" -D cookiejar.txt https://link/to/home/page

However, when I'm trying to use the generated cookiejar.txt file for subsequent calls, I'm not getting passed the authorization.

The browser sends the following data to the server:

GET /[my other page] HTTP/1.1
Host    [my host]
User-Agent  Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Firefox/31.0
Accept  text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language en-US,en;q=0.5
Accept-Encoding gzip, deflate
Cookie  JSESSIONID=[my session id]
Authorization   Basic [my encrypted string]
Connection  keep-alive

So, I changed my second cURL call to something like this, to be sure that all these parameters are sent as well:

curl -i -X GET -k -v \
-b cookiejar.txt \
-H "Authorization: Basic [my encrypted string]" \
-H "Host: [my host]"  \
-H "User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Firefox/31.0" \
-H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" \
-H "Content-Type: application/x-www-form-urlencoded" \
-H "Accept-Language: en-US,en;q=0.5" \
-H "Connection: Keep-Alive" \
https://[my other page]

Unfortunately this doesn't work. If I omit the Authorization header, I get a 401 error. If I include it in my cURL request, I get the Login page (with the 200 OK response).

There's no error in the console to give me at least a hint about what the problem is.

I appreciate any idea to help me get passed this issue.

  • Are you sure no Referer is needed? Have you tried with wget? eventhough wget is old I found bugs in cURL that doesn't allow me to access some https pages, but with wget I managed to get what I wanted. – YoMismo Feb 6 '15 at 7:57
  • I added a Referer as well. I just forgot to add it here. I will try with wget too, thank you for the suggestion. – Ionut Feb 6 '15 at 8:02
  • I just tried with wget too. It behaves the same as cURL. :( – Ionut Feb 6 '15 at 8:21
  • Have you tried debuging curl? Does it send the same headers as the ones the web browser is sending? – YoMismo Feb 6 '15 at 8:48
  • Yes, they are the same, except the JSESSIONID cookie and the Accept-Encoding header (gzip, deflate). The cookie is logical to be different. The encoding header cannot be used otherwise the server response gets unintelligible. – Ionut Feb 6 '15 at 9:27

It could be due to a redirect during authorization. See the -L and --location-trusted options in man curl. Also try testing -w redirect_url to see the actual page you would be redirected to if that is the case.

   -L, --location
          (HTTP/HTTPS) If the server reports that the requested page has  moved  to  a  different  location
          (indicated  with a Location: header and a 3XX response code), this option will make curl redo the
          request on the new place. If used together with -i, --include or -I,  --head,  headers  from  all
          requested  pages  will  be shown. When authentication is used, curl only sends its credentials to
          the initial host. If a redirect takes curl to a different host, it won't be able to intercept the
          user+password.  See  also  --location-trusted  on how to change this. You can limit the amount of
          redirects to follow by using the --max-redirs option.

          When curl follows a redirect and the request is not a plain GET (for example  POST  or  PUT),  it
          will  do  the  following  request  with  a  GET if the HTTP response was 301, 302, or 303. If the
          response code was any other 3xx code, curl will re-send the  following  request  using  the  same
          unmodified method.

          (HTTP/HTTPS)  Like  -L,  --location, but will allow sending the name + password to all hosts that
          the site may redirect to. This may or may not introduce a security breach if the  site  redirects
          you  to  a  site to which you'll send your authentication info (which is plaintext in the case of
          HTTP Basic authentication).

   -w, --write-out <format>
          Defines what to display on stdout after a completed and successful operation.  The  format  is  a
          string  that  may contain plain text mixed with any number of variables. The string can be speci‐
          fied as "string", to get read from a particular file you specify it "@filename" and to tell  curl
          to read the format from stdin you write "@-".

          The  variables  present  in  the output format will be substituted by the value or text that curl
          thinks fit, as described below. All variables are specified as %{variable_name} and to  output  a
          normal  % you just write them as %%. You can output a newline by using \n, a carriage return with
          \r and a tab space with \t.

          NOTE: The %-symbol is a special symbol in the win32-environment, where all occurrences of %  must
          be doubled when using this option.

          The variables available are:

          redirect_url   When  an  HTTP request was made without -L to follow redirects, this variable will
                         show the actual URL a redirect would take you to. (Added in 7.18.2)
  • Thank you for the answer. I added the -L and --location-trusted to my query but in vain. Nothing changed. I also used the -w option with this parameter: "Response code: %{http_code}\n Redirect URL: %{redirect_url}\n URL Effective: %{url_effective}". Unfortunately, nothing special got out of it. The results were: "Response code: 200" and the redirect URL was empty. The url_effective was exactly the url that I passed to curl. – Ionut Feb 6 '15 at 9:28
  • Oh, well that rules that out then :) – iyrin Feb 6 '15 at 9:33

I was able to reach the desired page in the end.

It seems that I wasn't following the correct sequence of URL calls. Once I did that, the desired page was retrieved correctly.

Thank you very much for the quick responses !

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.