Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a problem with the following wget command:

wget -nd -r -l 10 http://web.archive.org/web/20110726051510/http://feedparser.org/docs/

It should download recursively all of the linked documents on the original web but it downloads only two files (index.html and robots.txt).

How can I achieve recursive download of this web?

share|improve this question
up vote 17 down vote accepted

Wget by default honours the robots.txt standard for crawling pages, just like search engines do, and for archive.org, it disallows the entire /web/ subdirectory. To override, use -e robots= off,

wget -nd -r -l 10 -e robots=off http://web.archive.org/web/20110726051510/http://feedparser.org/docs/
share|improve this answer
Thank you. Is there some option to store every link only once? Maybe I should decrease 10 to lower number, but it's hard to guess. Now there is a file introduction.html, introduction.html.1, introduction.html.2 and I rather ended the process. – xralf Nov 25 '11 at 18:14
And the links are directing to the web. Is the --mirror option for the links to direct to the filesystem? – xralf Nov 25 '11 at 18:16
@xralf: well, you are using -nd, so different index.htmls are put in the same directory, and without -k, you'll not get rewriting of the links. – Ulrich Schwarz Nov 25 '11 at 18:35
$ wget --random-wait -r -p -e robots=off -U Mozilla \

Downloads recursively the content of the url.

--random-wait - wait between 0.5 to 1.5 seconds between requests.
-r - turn on recursive retrieving.
-e robots=off - ignore robots.txt.
-U Mozilla - set the "User-Agent" header to "Mozilla". Though a better choice is a real User-Agent like "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729)".

Some other useful options are:

--limit-rate=20k - limits download speed to 20kbps.
-o logfile.txt - log the downloads.
-l 0 - remove recursion depth (which is 5 by default).
--wait=1h - be sneaky, download one file every hour.
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.