Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am trying to mirror a blogger site so that I can have an exact copy of it on my filesystem to view. I have tried issuing the following command on Linux:

wget -r -k -x -e robots=off --wait 1 http://your.site.here.blogspot.com/

I have even tried using the -D flag to list a comma-separated list of domanins to follow (would prefer to just follow any domain though without having to specify all of them). I have even tried changing the .com part of the URL to the top-level domain for my country (.it) (without which for some reason I don't understand and would like to know, wget retrieves only index.html and no other page, perhaps someone here can explain why).

So, even when I do a

wget -r -k -x -e robots=off --wait 1 http://your.site.here.blogspot.it/

several HTML and also the favicon.ico are downloaded but none of the .png images from blogger are downloaded. Why is this so and how can I get wget to work properly. I've read the wget man page but had no luck.

Thanks.

share|improve this question
    
Are you sure the .png images are hosted on http://your.site.here.blogspot.it/? Images uploaded to the Blogger service seem to be served from <number>.bp.blogspot.com instead, which would explain why wget won't fetch them. –  jayhendren Oct 4 '13 at 0:38
add comment

2 Answers 2

Without error output of wget I can't tell what's the exact problem you have. But generally when downloading (or mirroring a website) with wget, I'll use the -mirror option like this:

wget --mirror -p --adjust-extension --wait 1 http://your.site.here.blogspot.it/
share|improve this answer
    
Well, from the man page the -p option takes a parameter so this can't be right. Also, -e robots=off is required since the site I am mirroring would otherwise disallow mirroring through the robots.txt file. The -wait 1 option is so the server does not get overloaded. However the image files do not get downloaded. Try this on any blogspot.com site and see for yourself. Where am I going wrong? –  John Sonderson Oct 4 '13 at 12:42
add comment

As jayhendren suggested, I had tried listing the domain bp.blogspot.com on the list following the -D flag. However what I forgot to do is add the -H flag. Why wget requires the extra -H flag to be added separately from the list of domains to follow with the -D flag is unclear to me, but it works. Here is the command I ultimately specified to mirror the Blogger site including the images served from the external domain:

wget --domains=blogspot.it,bp.blogspot.com -H --mirror -e robots=off \
  --wait 0.5 --convert-links http://yoursitehere.blogspot.it/

Note: this works from Italy. Convert .it to .com or to whatever other top-level domain if you want this to work from your location.

Regards.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.