Sign up ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

I would like to crawl links under and only download the links that is under

I am using the following wget command to get the files I want:

wget  -I ABC -r -e robots=off --wait 0.25

This works perfectly when I use wget 1.13.4. But the problem is I have to use this command on a server which has wget 1.11 and when I use the same command, it ends up downloading additional domains such as: 

How can I avoid this problem? I tried using


however it kept downloading those domains.

Also note that I can't use --no-parent since the files I want is in upper level (I want files under by crawling links under

Any hints?

share|improve this question
Don't use "fake" URLs, and don't use the URL of a site that has nothing to do with your problem. The domains you listed are in no way "subdomains", they are just different domains, probably belonging to the same company. – guntbert Oct 10 '13 at 19:44
Something is wrong here. wget should not cross hosts by default, and you need the -H/--span-hosts option to cross hosts when doing a recursive wget. "" is a completely different host from "". – jw013 Oct 10 '13 at 19:53
@guntbert sorry, i thought giving the real url would be a problem. of course, the website I want to crawl is not but when I crawl, I also see, in the main directory (the same level with – user2779485 Oct 10 '13 at 21:07
@jw013 like i said above, when i just give it also crawls, BUT this only happens with wget 1.11 not 1.13.. So that's why I am very confused. – user2779485 Oct 10 '13 at 21:09
This looks like a bug: wget 1.11 shouldn't behave differently, -H has always been required to recurse outside the original host. Does -D help? – Gilles Oct 10 '13 at 22:01

2 Answers 2

up vote 2 down vote accepted

This is wrong:


The right way is:


From the wget man page:

--exclude-domains domain-list
      Specify the domains that are not to be followed.
share|improve this answer
mustn't the list be komma separated? – rubo77 Jul 14 at 12:23
@rubo77 You're right, I've changed that. – Daniel Werner Sep 6 at 18:26

You may try --max-redirect 0 or use --domains as opposite of --exclude-domains


  -D,  --domains=LIST              comma-separated list of accepted domains.
       --exclude-domains=LIST      comma-separated list of rejected domains.
       --follow-tags=LIST          comma-separated list of followed HTML tags.
       --ignore-tags=LIST          comma-separated list of ignored HTML tags.
  -np, --no-parent                 don't ascend to the parent directory.
  --max-redirect                   maximum redirections allowed per page.
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.