I need to get a local copy of a website which requires confirming one's age before displaying content. No complex forms, just a checkbox and a button, "yes, I'm adult". I tried to use wget but with no success, it only downloads this initial screen and nothing more. Httrack failed, too.

I believe the confirmation info is stored in a cookie file, and I think I found out which one it is. So I copied it using the cookie.txt export chromium extension, then saved it to .txt and tried wget once again with --load-cookies option, but it still doesn't work.

How do I download website such as this one? Am I misusing wget or maybe there's a better way to do this?


OK, DownThemAll seems to do the work. Finally, I succeeded with wget as well, using

wget -mpkrl 0 http://example.com

maybe -rl 0 was what I was missing previously.

After investigating the page source I found that there's no reloading/redirecting involved, subpages are downloaded correctly and the confirmation screen is just an iframe on top of the page, so I can get rid of it with some simple script.

(In this particular case messing with cookies eventually was not necessary, so I'm not posting this as an answer.)

  • For anything all that advanced I would look into using curl instead of wget, curl can do some pretty complex stuff with forms and cookies. I would start with this curl tutorial and work your way up from that. – Bratchley May 29 '13 at 11:34
  • @JoelDavis, wget was enough this time, but thanks for the tutorial anyway, it seems very useful! – machaerus May 29 '13 at 12:28

It depends on how the site is set-up, not all of them use cookies for this - some may use things like php-sessions, and I don't know if wget can handle it directly. If you manage find the session-id (that part often gets hidden in the address-line by the browser or only used once as you log-in), you may be able to use that (the URL+session-id) as the start-url for wget.

Back to cookies... I have however needed to do similar things myself, so a few tips...

Some sites use cookies that are temporarily, ie. only last this session. These are usually not stored in the cookie.txt/cookies-database, so not all cookie-exporters will export them - if it exports from file/database, then no... if it exports from the browser-"memory" (on a site-base), then yes. Personally, I've found the "cookie.txt export" extension for the Chrome-browser the best, as it saves *all* cookies (including temps) from just the active tab. (well, actually it shows the cookies as text, you must mark&copy it and save it to file yourself) For a log-in; checking "Remember Me", will often turn a temporary cookie not stored in the cookie-file/database into a permanent one, so it can be easily exported (probably won't help for age-confirmation though).

After you've stored the cookies.txt file, it may be a good idea to edit it a bit - increase the expiration-time and perhaps editing the temp-cookies into permanent ones (think it's just a "switch"). You may also delete any non-related cookies (ie. ads and such).

Keeping the browser-window open while wget works - and perhaps refreshing the page or browsing a bit now and then - will make sure your session won't be expired before wget-finishes.

There are also extensions to browsers... I like "Down-them-all" for Firefox; which let you download content (though more interactively than wget). You get a list of links of the current page, and just check the links and/or images you want to save - and optionally, how many levels in depth you want to follow links. If you first log-in and/or check your age before starting it, then Down-them-all will be logged-in and checked-in together with your browser, and should be able to download all you could download manually. Play around with the renaming-patterns and numbering, as I think it'll store all pages flatly and there otherwise may be naming-conflict (it'll ask before overwriting though).

Good luck!

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.