I would like to crawl links under www.website.com/XYZ and only download the links that is under www.website.com/ABC.

I am using the following wget command to get the files I want:

wget  -I ABC -r -e robots=off --wait 0.25  http://www.website.com/XYZ

This works perfectly when I use wget 1.13.4. But the problem is I have to use this command on a server which has wget 1.11 and when I use the same command, it ends up downloading additional domains such as:


How can I avoid this problem? I tried using

--exclude domains=www.website.de,www.website.it

however it kept downloading those domains.

Also note that I can't use --no-parent since the files I want is in upper level (I want files under website.com/ABC by crawling links under website.com/XYZ).

Any hints?

  • Don't use "fake" URLs, and don't use the URL of a site that has nothing to do with your problem. The domains you listed are in no way "subdomains", they are just different domains, probably belonging to the same company. – guntbert Oct 10 '13 at 19:44
  • Something is wrong here. wget should not cross hosts by default, and you need the -H/--span-hosts option to cross hosts when doing a recursive wget. "www.website.com" is a completely different host from "www.website.de". – jw013 Oct 10 '13 at 19:53
  • @guntbert sorry, i thought giving the real url would be a problem. of course, the website I want to crawl is not website.com. but when I crawl example.com, I also see example.it, example.de in the main directory (the same level with example.com). – user2779485 Oct 10 '13 at 21:07
  • @jw013 like i said above, when i just give www.example.com/x it also crawls www.example.de, www.example.it. BUT this only happens with wget 1.11 not 1.13.. So that's why I am very confused. – user2779485 Oct 10 '13 at 21:09
  • This looks like a bug: wget 1.11 shouldn't behave differently, -H has always been required to recurse outside the original host. Does -D www.website.com help? – Gilles 'SO- stop being evil' Oct 10 '13 at 22:01

This is wrong:

--exclude domains=www.website.de,www.website.it

The right way is:

--exclude-domains www.website.de,www.website.it

From the wget man page:

--exclude-domains domain-list
      Specify the domains that are not to be followed.
  • mustn't the list be komma separated? – rubo77 Jul 14 '15 at 12:23
  • @rubo77 You're right, I've changed that. – Daniel Werner Sep 6 '15 at 18:26

You may try --max-redirect 0 or use --domains example.com as opposite of --exclude-domains example.com.


  -D,  --domains=LIST              comma-separated list of accepted domains.
       --exclude-domains=LIST      comma-separated list of rejected domains.
       --follow-tags=LIST          comma-separated list of followed HTML tags.
       --ignore-tags=LIST          comma-separated list of ignored HTML tags.
  -np, --no-parent                 don't ascend to the parent directory.
  --max-redirect                   maximum redirections allowed per page.
  • 5
    This is better than the accepted answer if you like white-listing the domains instead of black listing – yunzen Feb 10 '16 at 13:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.