Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I would like to crawl links under www.website.com/XYZ and only download the links that is under www.website.com/ABC.

I am using the following wget command to get the files I want:

wget  -I ABC -r -e robots=off --wait 0.25  http://www.website.com/XYZ

This works perfectly when I use wget 1.13.4. But the problem is I have to use this command on a server which has wget 1.11 and when I use the same command, it ends up downloading additional domains such as:

www.website.de 
www.website.it 
...

How can I avoid this problem? I tried using

--exclude domains=www.website.de,www.website.it

however it kept downloading those domains.

Also note that I can't use --no-parent since the files I want is in upper level (I want files under website.com/ABC by crawling links under website.com/XYZ).

Any hints?

share|improve this question
    
Don't use "fake" URLs, and don't use the URL of a site that has nothing to do with your problem. The domains you listed are in no way "subdomains", they are just different domains, probably belonging to the same company. –  guntbert Oct 10 '13 at 19:44
    
Something is wrong here. wget should not cross hosts by default, and you need the -H/--span-hosts option to cross hosts when doing a recursive wget. "www.website.com" is a completely different host from "www.website.de". –  jw013 Oct 10 '13 at 19:53
    
@guntbert sorry, i thought giving the real url would be a problem. of course, the website I want to crawl is not website.com. but when I crawl example.com, I also see example.it, example.de in the main directory (the same level with example.com). –  user2779485 Oct 10 '13 at 21:07
    
@jw013 like i said above, when i just give www.example.com/x it also crawls www.example.de, www.example.it. BUT this only happens with wget 1.11 not 1.13.. So that's why I am very confused. –  user2779485 Oct 10 '13 at 21:09
    
This looks like a bug: wget 1.11 shouldn't behave differently, -H has always been required to recurse outside the original host. Does -D www.website.com help? –  Gilles Oct 10 '13 at 22:01

2 Answers 2

up vote 0 down vote accepted

This is wrong:

--exclude domains=www.website.de,www.website.it

The right way is:

--exclude-domains www.website.de www.website.it

From man wget:

--exclude-domains domain-list
      Specify the domains that are not to be followed.
share|improve this answer

You may try --max-redirect 0 or use --domains example.com as opposite of --exclude-domains example.com.

See:

  -D,  --domains=LIST              comma-separated list of accepted domains.
       --exclude-domains=LIST      comma-separated list of rejected domains.
       --follow-tags=LIST          comma-separated list of followed HTML tags.
       --ignore-tags=LIST          comma-separated list of ignored HTML tags.
  -np, --no-parent                 don't ascend to the parent directory.
  --max-redirect                   maximum redirections allowed per page.
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.