wget doesn't convert URLs properly when --convert-links is specified

Question

According to the man pages, when I specify --convert-links to wget:

After the download is complete, convert the links in the document to make them suitable for local viewing [...] Note that only at the end of the download can Wget know which links have been downloaded. Because of that, the work done by -k will be performed at the end of all the downloads.

Let's say I do something like wget --convert-links http://stackoverflow.com. This site is large enough that I may never complete downloading everything. In this case, will I never get my links converted over?

When I run this locally from /tmp/wget, I see a directory called stackoverflow.com with files like index.html and tour.html. But when I load up the index page and hover over the link for the tour, it takes me to file:///tour.html, instead of file:///tmp/wget/stackoverflow.com/tour.html (the actual file location).

Answer 1

Your only question seems to be:

In this case, will I never get my links converted over?

I guess not. You could try this with a partition limited in size and see what wget does when it runs out of space (or use the --quota option, presumably it will still convert with that).

I can understand why this is the strategy. If this were done on the fly, then we would have to go through every document for every item that is downloaded. Consider:

• A.html is downloaded.
• B.html is downloaded, now check A.html for links.
• C.html is downloaded, now check A.html and B.html for links.
• ...and so on.

Instead, presumably a manifest is kept and then at the end all documents are scanned once. Of course, my on-the-fly method could be improved by keeping an index of all links, but I still think it will be orders of magnitude slower -- and require orders of magnitude more frequent writing to disk. Instead of saving a file, then changing it once at the end, you would commonly be rewriting it dozens or hundreds of times.