0

I've seen that in order to crawl an entire website, this command should work :

wget  --spider -r https://wikipedia.org/

But my question is why the same command to crawl an entire website, doesn't work with wikipedia ?

My goal is not to crawl all wikiepdia but to know the difference.

This is the output of the command:

Spider mode enabled. Check if remote file exists.
--2016-08-31 17:53:56--  http://wikipedia.org/
Resolving wikipedia.org (wikipedia.org)... 91.198.174.192, 2620:0:862:ed1a::1
Connecting to wikipedia.org (wikipedia.org)|91.198.174.192|:80... connected.
HTTP request sent, awaiting response... 301 TLS Redirect
Location: https://wikipedia.org/ [following]
Spider mode enabled. Check if remote file exists.
--2016-08-31 17:53:56--  https://wikipedia.org/
Connecting to wikipedia.org (wikipedia.org)|91.198.174.192|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.wikipedia.org/ [following]
Spider mode enabled. Check if remote file exists.
--2016-08-31 17:53:56--  https://www.wikipedia.org/
Resolving www.wikipedia.org (www.wikipedia.org)... 91.198.174.192, 2620:0:862:ed1a::1
Connecting to www.wikipedia.org (www.wikipedia.org)|91.198.174.192|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Remote file exists and could contain links to other resources -- retrieving.

--2016-08-31 17:53:56--  https://www.wikipedia.org/
Reusing existing connection to www.wikipedia.org:443.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘wikipedia.org/index.html’

    [ <=>                                                                                                                                                                                                                                   ] 81 292      --.-K/s   in 0,03s   

2016-08-31 17:53:57 (2,44 MB/s) - ‘wikipedia.org/index.html’ saved [81292]

Removing wikipedia.org/index.html.

Found no broken links.

FINISHED --2016-08-31 17:53:57--
Total wall clock time: 0,2s
Downloaded: 1 files, 79K in 0,03s (2,44 MB/s)
1

It's an FAQ (for both wget and Wikipedia):

By default, Wget plays the role of a web-spider that plays nice, and obeys a site's robots.txt file and no-follow attributes.

On 18 January 2005 the Google blog entry "Preventing comment spam" declared that Google would henceforth respect a rel="nofollow" attribute on hyperlinks. Their page ranking algorithm now ignores links with this attribute when ranking the destination page. The intended result is that site administrators can modify user-posted links such that the attribute is present, and thus an attempt to googlebomb by posting a link on such a site would yield no increase from that link.

The point is that Wikipedia has setup the site to discourage you from doing this.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.