3

In running wget, I would have expected it to "follow the links", as in depth-first: when it encounters a link, that is the next thing it fetches. But it seems that it first somehow processes the first page, gets all the links, and runs through those links. For each page it fetches, there may be new/nested links, but it doesn't get to those until it is done with the original page. Wanted to ask and see if this is in fact how it works, so I know if what I'm doing is working and it is just not fetching those pages until it's done with the starting page's links.

3

From the manual:

With HTTP URLs, Wget retrieves and parses the HTML or CSS from the given URL, retrieving the files the document refers to, through markup like href or src, or CSS URI values specified using the ‘url()’ functional notation. If the freshly downloaded file is also of type text/html, application/xhtml+xml, or text/css, it will be parsed and followed further.

Recursive retrieval of HTTP and HTML/CSS content is breadth-first. This means that Wget first downloads the requested document, then the documents linked from that document, then the documents linked by them, and so on. In other words, Wget first downloads the documents at depth 1, then those at depth 2, and so on until the specified maximum depth.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.