I'm trying to copy a forum thread with this directory structure:

The first page has a URL like this:


And the rest of the pages follow this format:


I'm using the command:

wget --recursive --page-requisites --adjust-extension --no-parent --convert-links https://some.site.com/foo/bar/threadNumber

This command can copy any single URL just fine. However, I want to put in the higher directory, and get all of the /page/* files as well. I want no higher directories, and nothing other than the lower /page/ files. I have also thrown --mirror into the mix with no success.

Any ideas why this command isn't going any lower to download the rest of the pages?

  • The -A and -R flags may be the key here...
    – Josh B.
    Feb 20 '17 at 21:41
  • I'm not posting this as an answer since you're mentioning wget, but have you considered one of the multitude of browser extensions designed to do exactly what you need? My favorite is Scrapbook, but I'm sure there are newer ones with fancier interface and all kinds of bells and whistles. Feb 27 '17 at 17:04
  • @DmitryGrigoryev I'm not sure a browser extension could provide the automation or power required for a project of this size, but, I think it would be worth looking up!
    – Josh B.
    Feb 27 '17 at 22:58

Recursively downloading links with a security check resulting in a loop redirection.

The command is not working because your website uses double redirection that confuses wget. Let's look at the verbose log that can be summoned with the --debug option (irrelevant lines removed):

---request begin---
---response begin---
HTTP/1.1 302 Found
Location: https://community.lego.com/auth/securityCheck?action=bounce&referrer=https%3A%2F%2Fcommunity.lego.com%2F<URL>
---request begin---
GET /auth/securityCheck?referrer=https%3A%2F%2Fcommunity.lego.com%2F<URL> HTTP/1.1
---response begin---
HTTP/1.1 302 Found
Location: https://community.lego.com/<URL>
Deciding whether to enqueue "https://community.lego.com/<URL>".
Already on the black list.
Decided NOT to load it.
Redirection "https://community.lego.com/<URL>" failed the test.

As seen, it bounces your request to some "security check" and back. Wget does not expect to be redirected second time to the same page you came from, and treats it as blacklisted, thus not following any links.

While it is absolutely possible there is a way to do some cookie magic to make the security check redirect not happen, I don't know how to do that.

However, if you are willing to recompile wget manually, the fix to your issue can be trivial: just add these 2 lines in src/recur.c.

            status = retrieve_url (url_parsed, url, &file, &redirected, referer,
                                   &dt, false, i, true);
+         if (redirected)
+                 hash_table_remove (blacklist, url);

            if (html_allowed && file && status == RETROK
                && (dt & RETROKF) && (dt & TEXTHTML))

That will remove the current page from the blacklist every time you are redirected, thus fixing your issue.

Be warned that it could possibly trigger infinite loops in some cases, so it's not a submission-ready patch.

Once you've rebuilt wget, you can just use something like wget -np -nd -r -k -p <url> to fetch your whole thread as intended.


Try this one:

wget -nv --mirror --span-hosts --convert-links --adjust-extension --page-requisites --no-parent https://some.site.com/foo/bar/threadNumber

This commands get the whole thread with all elements from all sites. You can limit it by types, --accept html,gif,png,jpg,jpeg for example.

  • I would need the images as well. Would -p do the trick?
    – Josh B.
    Feb 23 '17 at 22:43
  • Images hosted on other sites are not relative, so you must control the crawling with a bunch of -A und -R options. Tell me the site or type of board and I build the command for you.
    – ingopingo
    Feb 24 '17 at 8:23
  • I'm trying to back up some topics on the LEGO Message boards before they are taken offline on March 6th. Ideally I would like to backup all threads under this link. For example, I don't want anything higher than the "Classics" topic, but I do want every contained thread and all of their pages. Let me know if that doesn't make sense, thanks!
    – Josh B.
    Feb 24 '17 at 22:17
  • I tried your script, and it only downloads the given page as "index.html", with broken formatting. Running the command with -r, np, -p, and --convert-links also downloads the one page, but with proper formatting.
    – Josh B.
    Feb 26 '17 at 1:41
  • 1
    Hello Josh, I have try to fetch community.lego.com, but this site use AJAX technology. This task is too complex for wget, you need a full featured webcrawler with Javascript support, to parse, understand and follow the content.
    – ingopingo
    Feb 26 '17 at 9:55

Just thinking loud here.. what about something like webhttrack? or first run through jmeter -> export list of links -> then use wget or curl?

UPDATE Just tested with webhttrack, that one is working for the job and can been used both in CLI and Gui....

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.