18

I would like to crawl links under www.website.com/XYZ and only download the links that is under www.website.com/ABC.

I am using the following wget command to get the files I want:

wget  -I ABC -r -e robots=off --wait 0.25  http://www.website.com/XYZ

This works perfectly when I use wget 1.13.4. But the problem is I have to use this command on a server which has wget 1.11 and when I use the same command, it ends up downloading additional domains such as:

www.website.de 
www.website.it 
...

How can I avoid this problem? I tried using

--exclude domains=www.website.de,www.website.it

however it kept downloading those domains.

Also note that I can't use --no-parent since the files I want is in upper level (I want files under website.com/ABC by crawling links under website.com/XYZ).

Any hints?

7
  • Don't use "fake" URLs, and don't use the URL of a site that has nothing to do with your problem. The domains you listed are in no way "subdomains", they are just different domains, probably belonging to the same company.
    – guntbert
    Oct 10, 2013 at 19:44
  • Something is wrong here. wget should not cross hosts by default, and you need the -H/--span-hosts option to cross hosts when doing a recursive wget. "www.website.com" is a completely different host from "www.website.de".
    – jw013
    Oct 10, 2013 at 19:53
  • @guntbert sorry, i thought giving the real url would be a problem. of course, the website I want to crawl is not website.com. but when I crawl example.com, I also see example.it, example.de in the main directory (the same level with example.com). Oct 10, 2013 at 21:07
  • @jw013 like i said above, when i just give www.example.com/x it also crawls www.example.de, www.example.it. BUT this only happens with wget 1.11 not 1.13.. So that's why I am very confused. Oct 10, 2013 at 21:09
  • This looks like a bug: wget 1.11 shouldn't behave differently, -H has always been required to recurse outside the original host. Does -D www.website.com help? Oct 10, 2013 at 22:01

2 Answers 2

27

You may try --max-redirect 0 or use --domains example.com as opposite of --exclude-domains example.com.

See:

  -D,  --domains=LIST              comma-separated list of accepted domains.
       --exclude-domains=LIST      comma-separated list of rejected domains.
       --follow-tags=LIST          comma-separated list of followed HTML tags.
       --ignore-tags=LIST          comma-separated list of ignored HTML tags.
  -np, --no-parent                 don't ascend to the parent directory.
  --max-redirect                   maximum redirections allowed per page.
1
  • 8
    This is better than the accepted answer if you like white-listing the domains instead of black listing
    – yunzen
    Feb 10, 2016 at 13:12
10

This is wrong:

--exclude domains=www.website.de,www.website.it

The right way is:

--exclude-domains www.website.de,www.website.it

From the wget man page:

--exclude-domains domain-list
      Specify the domains that are not to be followed.
2
  • mustn't the list be komma separated?
    – rubo77
    Jul 14, 2015 at 12:23
  • @rubo77 You're right, I've changed that. Sep 6, 2015 at 18:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.