Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am trying to learn how to use recursive download with wget from the wget info page.

For example, let us try to download all the images of xkcd. A list of all the pages is present in the xkcd archive. All pages have a single png file in them. The png file is present in a different host, imgs.xkcd.com.

I tried with this command:

wget -r -HD imgs.xkcd.com -l 2 -A.png http://www.xkcd.com/archive/ --random-wait

The result:

 xkcd $ tree
.

0 directories, 0 files

 xkcd $ wget -r -HD imgs.xkcd.com -l 2 -A.png http://www.xkcd.com/archive/ --random-wait
--2014-01-10 18:49:55--  http://www.xkcd.com/archive/
Resolving www.xkcd.com (www.xkcd.com)... 107.6.106.82
Connecting to www.xkcd.com (www.xkcd.com)|107.6.106.82|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 83226 (81K) [text/html]
Saving to: `www.xkcd.com/archive/index.html'

100%[=============================================================================================================>] 83,226      68.3K/s   in 1.2s    

2014-01-10 18:49:57 (68.3 KB/s) - `www.xkcd.com/archive/index.html' saved [83226/83226]

Loading robots.txt; please ignore errors.
--2014-01-10 18:49:57--  http://imgs.xkcd.com/robots.txt
Resolving imgs.xkcd.com (imgs.xkcd.com)... 107.6.106.82
Reusing existing connection to www.xkcd.com:80.
HTTP request sent, awaiting response... 404 Not Found
2014-01-10 18:49:58 ERROR 404: Not Found.

Removing www.xkcd.com/archive/index.html since it should be rejected.

--2014-01-10 18:49:58--  http://imgs.xkcd.com/static/terrible_small_logo.png
Reusing existing connection to www.xkcd.com:80.
HTTP request sent, awaiting response... 200 OK
Length: 11001 (11K) [image/png]
Saving to: `imgs.xkcd.com/static/terrible_small_logo.png'

100%[=============================================================================================================>] 11,001      --.-K/s   in 0.05s   

2014-01-10 18:49:58 (229 KB/s) - `imgs.xkcd.com/static/terrible_small_logo.png' saved [11001/11001]

FINISHED --2014-01-10 18:49:58--
Total wall clock time: 2.9s
Downloaded: 2 files, 92K in 1.2s (74.4 KB/s)

 xkcd $ tree
.
|-- imgs.xkcd.com
|   `-- static
|       `-- terrible_small_logo.png
`-- www.xkcd.com
    `-- archive

4 directories, 1 file

 xkcd $

This is obviously not what I want. It seems that wget rejected www.xkcd.com/archive/index.html before reading it and checking for links. Even if .html is added to the accept list (as suggested in an answer), it doesn't download the images. What is the mistake in the command?

share|improve this question
add comment

1 Answer

up vote 2 down vote accepted

The problem is your restriction on which links to follow. You've set it to only follow links to imgs.xkcd.com. But the /archive/ page doesn't contain any links directly there - it only contains links to other pages on www.xkcd.com, and then those pages contain the link to imgs.xkcd.com.

So you will need to allow that domain, too. This command works:

wget -r -HD imgs.xkcd.com,www.xkcd.com -l 2 -A.png http://www.xkcd.com/archive/ --random-wait
share|improve this answer
    
Its seems that this command does not do the trick. You can check it yourself. –  Kartik Jan 10 at 13:38
    
Sorry. Correcting my answer now... –  Jenny D Jan 10 at 13:43
    
There - now it works. Also I've now got a complete xkcd archive on my own computer thanks to this question :-) –  Jenny D Jan 10 at 13:53
    
Thanks. Teh command worked. But now I've got a lot of empty folders, how to remove them? –  Kartik Jan 10 at 14:09
    
rm -rf www.xkcd.com; mv imgs.xkcd.com/comics xkcd-comics; rm -rf imgs.xkcd.com –  Jenny D Jan 10 at 14:16
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.