1

Back to basics. I have a static web where the links are typically relative.

All works fine in browsers and in Google Crawler, but wget --spider resolve relative links against the site root.

That is, in a page at /foo/bar.html, a link <a href="test.html"> is resolved as /test.html, instead of /foo/test.html.

My question is: Did I forget/overlook something basic in how HTML links work, or is wget --spider resolving the URIs wrong?

The HTML specification says that if there is no base element, then the current document is the base.

There's this wget issue but it should be fixed in my version.

$ wget --version
GNU Wget 1.19.4 built on linux-gnu.

-cares +digest -gpgme +https +ipv6 +iri +large-file -metalink +nls 
+ntlm +opie +psl +ssl/openssl

My command is:

wget --spider --convert-links -r -p --level 1 -E --user-agent=crawlerTest http://www.zizka.ch/
0

I generally use these switches when I want to spider a site using wget:

$ wget -r -l4 –spider -D unix.stackexchange.com https://unix.stackexchange.com/

This tells wget to recursively (-r) spider (-spider) up to 4 levels deep (-l4). The -D switch tells wget to only follow links which fall in this domain.

Running this will look like this:

$ timeout 1 wget -r -l4 –spider -D unix.stackexchange.com https://unix.stackexchange.com/
--2018-07-31 20:28:40--  http://xn--spider-vg0c/
Resolving xn--spider-vg0c (xn--spider-vg0c)... failed: nodename nor servname provided, or not known.
wget: unable to resolve host address ‘xn--spider-vg0c’
--2018-07-31 20:28:40--  https://unix.stackexchange.com/
Resolving unix.stackexchange.com (unix.stackexchange.com)... 151.101.65.69, 151.101.193.69, 151.101.129.69, ...
Connecting to unix.stackexchange.com (unix.stackexchange.com)|151.101.65.69|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 106032 (104K) [text/html]
Saving to: ‘unix.stackexchange.com/index.html’

unix.stackexchange.com/index.html               100%[====================================================================================================>] 103.55K  --.-KB/s    in 0.1s

2018-07-31 20:28:40 (1.02 MB/s) - ‘unix.stackexchange.com/index.html’ saved [106032/106032]

Loading robots.txt; please ignore errors.
--2018-07-31 20:28:40--  https://unix.stackexchange.com/robots.txt
Reusing existing connection to unix.stackexchange.com:443.
HTTP request sent, awaiting response... 200 OK
Length: 2148 (2.1K) [text/plain]
Saving to: ‘unix.stackexchange.com/robots.txt’

unix.stackexchange.com/robots.txt               100%[====================================================================================================>]   2.10K  --.-KB/s    in 0s

2018-07-31 20:28:40 (228 MB/s) - ‘unix.stackexchange.com/robots.txt’ saved [2148/2148]
  • 1
    That's nice, but does any of that affect relative links? – Ondra Žižka Aug 1 '18 at 3:13
  • I'm not able to replicate your issue locally. Can you point to a specific spot where you're seeing an issue? – slm Aug 1 '18 at 8:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.