19

I would like to crawl links under www.website.com/XYZ and only download the links that is under www.website.com/ABC.

I am using the following wget command to get the files I want:

wget  -I ABC -r -e robots=off --wait 0.25  http://www.website.com/XYZ

This works perfectly when I use wget 1.13.4. But the problem is I have to use this command on a server which has wget 1.11 and when I use the same command, it ends up downloading additional domains such as:

www.website.de 
www.website.it 
...

How can I avoid this problem? I tried using

--exclude domains=www.website.de,www.website.it

however it kept downloading those domains.

Also note that I can't use --no-parent since the files I want is in upper level (I want files under website.com/ABC by crawling links under website.com/XYZ).

Any hints?

7
  • Don't use "fake" URLs, and don't use the URL of a site that has nothing to do with your problem. The domains you listed are in no way "subdomains", they are just different domains, probably belonging to the same company.
    – guntbert
    Oct 10, 2013 at 19:44
  • Something is wrong here. wget should not cross hosts by default, and you need the -H/--span-hosts option to cross hosts when doing a recursive wget. "www.website.com" is a completely different host from "www.website.de".
    – jw013
    Oct 10, 2013 at 19:53
  • @guntbert sorry, i thought giving the real url would be a problem. of course, the website I want to crawl is not website.com. but when I crawl example.com, I also see example.it, example.de in the main directory (the same level with example.com). Oct 10, 2013 at 21:07
  • @jw013 like i said above, when i just give www.example.com/x it also crawls www.example.de, www.example.it. BUT this only happens with wget 1.11 not 1.13.. So that's why I am very confused. Oct 10, 2013 at 21:09
  • This looks like a bug: wget 1.11 shouldn't behave differently, -H has always been required to recurse outside the original host. Does -D www.website.com help? Oct 10, 2013 at 22:01

2 Answers 2

27

You may try --max-redirect 0 or use --domains example.com as opposite of --exclude-domains example.com.

See:

  -D,  --domains=LIST              comma-separated list of accepted domains.
       --exclude-domains=LIST      comma-separated list of rejected domains.
       --follow-tags=LIST          comma-separated list of followed HTML tags.
       --ignore-tags=LIST          comma-separated list of ignored HTML tags.
  -np, --no-parent                 don't ascend to the parent directory.
  --max-redirect                   maximum redirections allowed per page.
1
  • 8
    This is better than the accepted answer if you like white-listing the domains instead of black listing
    – yunzen
    Feb 10, 2016 at 13:12
10

This is wrong:

--exclude domains=www.website.de,www.website.it

The right way is:

--exclude-domains www.website.de,www.website.it

From the wget man page:

--exclude-domains domain-list
      Specify the domains that are not to be followed.
2
  • mustn't the list be komma separated?
    – rubo77
    Jul 14, 2015 at 12:23
  • @rubo77 You're right, I've changed that. Sep 6, 2015 at 18:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .