5

I am trying to learn how to use recursive download with wget from the wget info page.

For example, let us try to download all the images of xkcd. A list of all the pages is present in the xkcd archive. All pages have a single png file in them. The png file is present in a different host, imgs.xkcd.com.

I tried with this command:

wget -r -HD imgs.xkcd.com -l 2 -A.png http://www.xkcd.com/archive/ --random-wait

The result:

 xkcd $ tree
.

0 directories, 0 files

 xkcd $ wget -r -HD imgs.xkcd.com -l 2 -A.png http://www.xkcd.com/archive/ --random-wait
--2014-01-10 18:49:55--  http://www.xkcd.com/archive/
Resolving www.xkcd.com (www.xkcd.com)... 107.6.106.82
Connecting to www.xkcd.com (www.xkcd.com)|107.6.106.82|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 83226 (81K) [text/html]
Saving to: `www.xkcd.com/archive/index.html'

100%[=============================================================================================================>] 83,226      68.3K/s   in 1.2s    

2014-01-10 18:49:57 (68.3 KB/s) - `www.xkcd.com/archive/index.html' saved [83226/83226]

Loading robots.txt; please ignore errors.
--2014-01-10 18:49:57--  http://imgs.xkcd.com/robots.txt
Resolving imgs.xkcd.com (imgs.xkcd.com)... 107.6.106.82
Reusing existing connection to www.xkcd.com:80.
HTTP request sent, awaiting response... 404 Not Found
2014-01-10 18:49:58 ERROR 404: Not Found.

Removing www.xkcd.com/archive/index.html since it should be rejected.

--2014-01-10 18:49:58--  http://imgs.xkcd.com/static/terrible_small_logo.png
Reusing existing connection to www.xkcd.com:80.
HTTP request sent, awaiting response... 200 OK
Length: 11001 (11K) [image/png]
Saving to: `imgs.xkcd.com/static/terrible_small_logo.png'

100%[=============================================================================================================>] 11,001      --.-K/s   in 0.05s   

2014-01-10 18:49:58 (229 KB/s) - `imgs.xkcd.com/static/terrible_small_logo.png' saved [11001/11001]

FINISHED --2014-01-10 18:49:58--
Total wall clock time: 2.9s
Downloaded: 2 files, 92K in 1.2s (74.4 KB/s)

 xkcd $ tree
.
|-- imgs.xkcd.com
|   `-- static
|       `-- terrible_small_logo.png
`-- www.xkcd.com
    `-- archive

4 directories, 1 file

 xkcd $

This is obviously not what I want. It seems that wget rejected www.xkcd.com/archive/index.html before reading it and checking for links. Even if .html is added to the accept list (as suggested in an answer), it doesn't download the images. What is the mistake in the command?

Improve this question

2 Answers 2

Reset to default
3

The problem is your restriction on which links to follow. You've set it to only follow links to imgs.xkcd.com. But the /archive/ page doesn't contain any links directly there - it only contains links to other pages on www.xkcd.com, and then those pages contain the link to imgs.xkcd.com.

So you will need to allow that domain, too. This command works:

wget -r -HD imgs.xkcd.com,www.xkcd.com -l 2 -A.png http://www.xkcd.com/archive/ --random-wait
Improve this answer
5
  • Its seems that this command does not do the trick. You can check it yourself.
    – Kartik
    Jan 10, 2014 at 13:38
  • Sorry. Correcting my answer now...
    – Jenny D
    Jan 10, 2014 at 13:43
  • There - now it works. Also I've now got a complete xkcd archive on my own computer thanks to this question :-)
    – Jenny D
    Jan 10, 2014 at 13:53
  • Thanks. Teh command worked. But now I've got a lot of empty folders, how to remove them?
    – Kartik
    Jan 10, 2014 at 14:09
  • rm -rf www.xkcd.com; mv imgs.xkcd.com/comics xkcd-comics; rm -rf imgs.xkcd.com
    – Jenny D
    Jan 10, 2014 at 14:16
0

I think the problem sometimes could be also with:

Reusing existing connection to www.xkcd.com:80.

HTTP request sent, awaiting response... 200 OK

which indicates the files weren't downloaded, but they were re-used.

So I think removal of the existing files in some cases such as index.html (which could be breaking recursive download going further) would help as well.

Or basically starting it from scratch (without -c).

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .