GNU Wget 1.16 built on linux-gnueabihf on Raspberry Pi 3

How do I force wget to get entire site (follow links, act like robot), not only first index?

I tried:

wget -r http://aol.com
wget -r -l0 http://aol.com
wget -r -m -l0 http://aol.com

Every command finished with the same:

--2017-11-29 08:05:42--  http://aol.com/
Resolving aol.com (aol.com)...,,, ...
Connecting to aol.com (aol.com)||:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.aol.com/ [following]
--2017-11-29 08:05:42--  https://www.aol.com/
Resolving www.aol.com (www.aol.com)...,,, ...
Connecting to www.aol.com (www.aol.com)||:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Last-modified header missing -- time-stamps turned off.
--2017-11-29 08:05:44--  https://www.aol.com/
Reusing existing connection to www.aol.com:443.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘aol.com/index.html’

aol.com/index.html                                              [  <=>                                                                                                                                      ] 359.95K   751KB/s   in 0.5s

2017-11-29 08:05:45 (751 KB/s) - ‘aol.com/index.html’ saved [368585]

FINISHED --2017-11-29 08:05:45--
Total wall clock time: 2.8s
Downloaded: 1 files, 360K in 0.5s (751 KB/s)

What am I doing wrong?


Your problem arises because all links on aol.com/index.html point to different hosts. To recursively download from all hosts you can add the option --span-hosts. To allow all aol hosts it seemed to work for me to add the --span-hosts '*.aol.com' option.

wget --span-hosts '*.aol.com' -r http://www.aol.com

You can list the links with

grep -Po '(?<=href=")[^"]*' aol.com/index.html

You'll see the most of them point to www.aol.com so you could also call

wget -r http://www.aol.com
| improve this answer | |
  • The grep is just running on the file you downloaded in the question, so make sure you are in the right directory and haven't deleted it. – Mark Perryman Nov 29 '17 at 8:43

Using the following will allow wget to download all pages linked to an a website recursively.

wget -r $(curl http://aol.com | grep -Po '(?<=href=")[^"]*')

Replace the example website with the one you want. This will fetch like Deapth for search in a graph

Method of Working curl will fetch index.html. It'll be piped into grep to find all links by matching href. The enter result will be given to wget as a variable. wget fetch the links one by one from the variable.

| improve this answer | |
  • 1
    well that will only work for 1 level of recursion :-( – Flash Thunder Nov 29 '17 at 13:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.