Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Before I start the question off, I want to say that a similar question helped me get past the initial login. My issue is as stated below.

There's a website that I'm trying to mirror. It is something that I have an account for. I'm using wget as my tool of choice. I tried curl, but found that while submitting post data is easy with it, wget is better equipped for the task at hand.

The website has an initial login page that it redirects to. After this, you have access to everything on the website. Logins do timeout after so long, but that's it.

With the wget commands below, I was able to successfully save my cookies, load them, and download all child folders. My issue, however, is that each child has an index.html of the same login page. It's like the cookie worked fine for the root folder but nothing beneath it.

The commands I used were:

wget http://site.here.com/users/login --save-cookies cookies.txt --post-data 'email=example@test.com&password=*****&remember_me=1' --keep-session-cookies --delete-after
wget http://site.here.com/ --load-cookies cookies.txt --keep-session-cookies -r -np

Note that the post-data variables/ids are different and that I had to download the login page to see what they were.

Secondly, note that if I didn't put remember_me value to 1 that cookies.txt would be different.

Without remember_me=1

.here.com    TRUE    /   FALSE   numbershere   CAKEPHP   garbagehere

With remember_me=1

site.here.com  FALSE   /   FALSE   numbershere   CakeCookie[rememberme]   garbage
.here.com    TRUE    /   FALSE   numbershere   CAKEPHP   garbagehere

The result being that the former would only download the login page and the latter getting to all child folders, only with children containing index of login and that's it.

I'm kind of stuck and my experience with wget and http is very limited. What would you do to get past this? Generate a cookie for each child? How would you automate that instead of manually creating a cookie file for each child?

P.S: I'm using Linux if that reflects the answers I'm given.

share|improve this question

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.