Sign up ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

I am trying to recursively download, and am getting exactly one file, the root index.html, downloaded.

I've looked at Download recursively with wget and started using the recommended -e robots=off, but it still behaves the same.

How, with wget or some other tool, can I download a copy of the site?

share|improve this question
Possible duplicate of this question? – Geeb Jan 27 '14 at 16:12

2 Answers 2

up vote 5 down vote accepted

you are asking wget to do a recursive download of, but this URL doesn't provide any direct content. instead it is just a re-direct to (which you haven't told wget to fetch recursively)..

if you tell wget to download the correct URL it will work:

wget -r -e robots=off http://www....
share|improve this answer

It's because wget defaults to only doing recursive download within the hostname that you used when you started. issues a redirect to Since all further links are under, wget will consider those links as being off-site and won't follow them.

The easiest solution here is of course to start with wget -r

You could also add to the list of domains to follow:

wget -r -D

For the future, you can find this kind of information by adding the debug flag. When I did that, I got

Deciding whether to enqueue "".
This is not the same hostname as the parent's ( and
Decided NOT to load it.
Redirection "" failed the test.
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.