wget -c website's certificate verification

Question

anisha@linux-dopx:~> wget -c --no-check-certificate --user abc --password xyz https://www.smart-developer.com/lnmshop/download/SmartDeveloperUS/203/SD_02.pdf

--2011-08-29 15:11:53--  https://www.smart-developer.com/lnmshop/download/SmartDeveloperUS/203/SD_02.pdf
Resolving www.smart-developer.com... 80.237.227.161
Connecting to www.smart-developer.com|80.237.227.161|:443... connected.
WARNING: cannot verify www.smart-developer.com's certificate, issued by `/C=US/O=Thawte, Inc./OU=Domain Validated SSL/CN=Thawte DV SSL CA':
  Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `SD_02.pdf'
    [   <=>                                                                   ] 19,286      10.4K/s   in 1.8s    
2011-08-29 15:12:00 (10.4 KB/s) - `SD_02.pdf' saved [19286]

The book that gets downloaded is in KB (actual size 20 MB), and shows error on opening too, where as on downloading through Firefox directly, I have no issues.

EDIT 1: Seems the shell can't tolerate @ in the username!

See the placements of quotes too.

Scroll right.

anisha@linux-dopx:~> wget -c https://x.y.z@gmail.com:abc@www.smart-developer.com/lnmshop/download/SmartDeveloperUS/203/SD_02.pdf
https://x.y.z@gmail.com:abc@www.smart-developer.com/lnmshop/download/SmartDeveloperUS/203/SD_02.pdf: Bad port number.

anisha@linux-dopx:~> wget -c https://'x.y.z@gmail.com':abc@www.smart-developer.com/lnmshop/download/SmartDeveloperUS/203/SD_02.pdf
https://x.y.z@gmail.com:abc@www.smart-developer.com/lnmshop/download/SmartDeveloperUS/203/SD_02.pdf: Bad port number.

anisha@linux-dopx:~> wget -c https://'x.y.z@gmail.com:abc'@www.smart-developer.com/lnmshop/download/SmartDeveloperUS/203/SD_02.pdf
https://x.y.z@gmail.com:abc@www.smart-developer.com/lnmshop/download/SmartDeveloperUS/203/SD_02.pdf: Bad port number.

anisha@linux-dopx:~> wget -c https://x.y.z:abc@www.smart-developer.com/lnmshop/download/SmartDeveloperUS/203/SD_02.pdf
--2011-08-29 16:01:38--  https://x.y.z:*password*@www.smart-developer.com/lnmshop/download/SmartDeveloperUS/203/SD_02.pdf
Resolving www.smart-developer.com... 80.237.227.161
Connecting to www.smart-developer.com|80.237.227.161|:443... connected.
ERROR: cannot verify www.smart-developer.com's certificate, issued by `/C=US/O=Thawte, Inc./OU=Domain Validated SSL/CN=Thawte DV SSL CA':
  Unable to locally verify the issuer's authority.
To connect to www.smart-developer.com insecurely, use `--no-check-certificate'.
Unable to establish SSL connection.
anisha@linux-dopx:~/Desktop>

Answer 1

Authentication Types

HTTP supports authentication natively, and that's what you've used. The native HTTP authentication is rarely used, however. HTTP also allows form data to be sent to a server; most websites, including the one you're trying to work with, use this form-based authentication instead of HTTP authentication because it is more user-friendly and customizable.

How form authentication works

When an HTTP client requests a resource from an HTTP server, lots of data is sent in the HTTP header. The important part for form-based authentication is the cookies. A cookie is just a name for a set of key=value pairs. Several cookies can be sent in an HTTP request.

When a request for a restricted resource comes in, the server will look for a specific authentication token in the session cookie. If it doesn't exist, it means the client has not logged in, so it returns a login form instead of the requested resource. If the token does exist and is valid, it means the client has logged in, so the server returns the resource that was requested.

So, how do we get that special authentication token, and how do we send it? We get it by submitting the login form with valid data. Once the server authenticates the client, it sends back a cookie in the HTTP header. We need to save that cookie and pass it in with all future requests.

The brass tacks

Information Gathering

View the source for the login page. You already downloaded it using wget, just open that in a text editor. Look for <form...> in the source. In general, there may be several, and you might have to figure out which one is for submitting the login data. In the one you're working with now, there is just one form. Next look for all the <input...> tags. These have data that will be submitted as part of the form data. Of particular interest is the name=... part. This is the key part of the key=value pair, and whatever you fill in is the value part.

The page you're working with has the following fields Login, Password, AdminSiteaccessURI, LoginButton, and RedirectURI. The </form> is the end of the form - you don't have to look any further. We need to submit all those keys with the appropriate values. Some of the values are filled in by you - that includes Login and Password, others, you can take the value that is already present in the field. If you look at the RedirectURI, you'll see it has a value= attribute the server filled in the URI for the actual resource you were requesting in the first place.

There is one more key piece of information: where and how do we submit the data? This can be found in the <form...> tag. It will have a method and an action. The action doesn't have the full URL; it has a URL relative to the base, but you'll need the full URL, so add the base to the relative URL. The action is the URL to submit to, and the action lets you know if its a POST or a GET request. The page you're working with needs the data submitted via the POST method.

Submiting the login data

Now that we know what we're going to submit, we need to actually do it. We can use wget for this. If you look at the man page, you'll find a --post-data=string option; it shows you how to format it: key1=value1&key2=value2.... It's simply key=value for each pair, concatenated by a &. This is not the only option we need.

Remember the authentication token? The server will send it back as part of the response, and we need to save it. Looking through the man page again for "cookie", you'll see a couple of important options. --save-cookies file and --keep-session-cookies.

Build up your command line with those options and execute it, keeping in mind the URL you're submitting to is not the resource you were originally trying to get - you need to submit to the URL pointed to by the <form action=...> attribute. You should end up with your session cookies in a file, and possibly (if wget successfully follows HTTP redirects), the PDF file as the output. If it didn't get the PDF file, you'll just submit another request (this time without any --post-data), but this time, include the cookies you saved before using `--load-cookies file'.

That should do it!