I would like to download all the books from the gutenberg project using wget. I want to get them in the epub format. The link to the list is: http://www.gutenberg.org/robot/harvest?filetypes[]=epub.images&langs[]=fr but when i run:

  wget -H -w 2 -m "http://www.gutenberg.org/robot/harvest?filetypes[]=epub.images&langs[]=fr"

It downloads the html page not all the files linked in the page. Hope this is clear enough. It is inspired from this question: https://webapps.stackexchange.com/questions/12311/how-to-download-all-english-books-from-gutenberg


You're doing everything right. It's Gutenberg's servers that are broken (temporarily, I hope).

What happens is: the pages you're fetching link to EPUB files under aleph.gutenberg.org/cache/, but aleph.gutenberg.org disallows robot access to the /cache. That's why Wget skips all the EPUB files.

I wrote an email to Project Gutenberg, and I hope they'll fix this soon — their contact page promises answers within two days. Just try again in a few days. (Or find a way to work around robots.txt. I don't want to publish it here because most of the time people misuse it to do bad stuff.)

Update November 29th: the problem have been fixed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.