Is there any way to get wget to recursively download from one starting URL (including every prerequisites) and include other paths for which the same will be done?

To make this clearer: The file index.html in http://www.domain1.com/path1/index.html links to several images, style sheets etc. This is what the -p option is for. But what if index.html links to http://www.domain2.com/path2/site.html which I also want to download (of course including its prerequisites, too)? If this second site site.html would link to other sites they should not be included.

I have already tried to add the second path via the --accept-regex option but this didn't result in the desired behaviour.

  • Have you tried adding the -r -l 1 options? FWIW, the wget docs suggest a few extra options that are useful when using -p: wget -E -H -k -K -p http://SITE/DOCUMENT; please see the wget man page for details. – PM 2Ring Dec 22 '14 at 12:19

There are a few different restrictions you can place on wget to achieve what you're trying to accomplish, using your example.

Related options:

-D allows you to provide a comma-separated list of accepted domains - for example, -D domain1.com,domain2.com would ensure you only download files from those two domains.

-l allows you to specify the maximum recursion depth - using your own explanation, an example would be -l 1, which would mean if domain1's index.html lead to domain2's site.html, site.html would be downloaded, but nothing that site.html links to. Although -l is rather restrictive, you can run wget multiple times with different restrictions to obtain all of the files you require.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy