Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I want to mirror one of my websites.

Unfortunately, the site is in its preview mode, and a GET param must be appended to all URLs in the page for it to continue in its preview mode.

This GET param was added with JavaScript. Obviously, that does not affect wget.

Is there a way I can send the returned HTML off to a script to add these myself before wget attempts to crawl all the URLs?

share|improve this question
add comment

1 Answer

up vote 1 down vote accepted

Your question is not very clear. What I understand is that:

  • you're retrieving web pages recursively;
  • your pages contain internal links;
  • following these links leads to a non-existing page, and you would like to rewrite each link to add an extra parameter to the HTTP request.

I don't think you can do that with wget. I suggest using LWP instead. It comes with a script called lwp-rget that performs recursive retrievals. There's a line

my $req = HTTP::Request->new(GET => $url);

Just before this line, add some code to mangle the URL as appropriate, something like

$url->query_form($url->query_form, "show_preview" => "yes");
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.