1

GNU Wget 1.16 built on linux-gnueabihf on Raspberry Pi 3

How do I force wget to get entire site (follow links, act like robot), not only first index?

I tried:

wget -r http://aol.com
wget -r -l0 http://aol.com
wget -r -m -l0 http://aol.com

Every command finished with the same:

--2017-11-29 08:05:42--  http://aol.com/
Resolving aol.com (aol.com)... 149.174.149.73, 64.12.249.135, 149.174.110.105, ...
Connecting to aol.com (aol.com)|149.174.149.73|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.aol.com/ [following]
--2017-11-29 08:05:42--  https://www.aol.com/
Resolving www.aol.com (www.aol.com)... 34.233.220.13, 34.235.7.32, 52.6.64.98, ...
Connecting to www.aol.com (www.aol.com)|34.233.220.13|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Last-modified header missing -- time-stamps turned off.
--2017-11-29 08:05:44--  https://www.aol.com/
Reusing existing connection to www.aol.com:443.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘aol.com/index.html’

aol.com/index.html                                              [  <=>                                                                                                                                      ] 359.95K   751KB/s   in 0.5s

2017-11-29 08:05:45 (751 KB/s) - ‘aol.com/index.html’ saved [368585]

FINISHED --2017-11-29 08:05:45--
Total wall clock time: 2.8s
Downloaded: 1 files, 360K in 0.5s (751 KB/s)

What am I doing wrong?

2 Answers 2

2

Your problem arises because all links on aol.com/index.html point to different hosts. To recursively download from all hosts you can add the option --span-hosts. To allow all aol hosts it seemed to work for me to add the --span-hosts '*.aol.com' option.

wget --span-hosts '*.aol.com' -r http://www.aol.com

You can list the links with

grep -Po '(?<=href=")[^"]*' aol.com/index.html

You'll see the most of them point to www.aol.com so you could also call

wget -r http://www.aol.com
1
  • The grep is just running on the file you downloaded in the question, so make sure you are in the right directory and haven't deleted it. Nov 29, 2017 at 8:43
0

Using the following will allow wget to download all pages linked to an a website recursively.

wget -r $(curl http://aol.com | grep -Po '(?<=href=")[^"]*')

Replace the example website with the one you want. This will fetch like Deapth for search in a graph

Method of Working curl will fetch index.html. It'll be piped into grep to find all links by matching href. The enter result will be given to wget as a variable. wget fetch the links one by one from the variable.

1
  • 1
    well that will only work for 1 level of recursion :-( Nov 29, 2017 at 13:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .