Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I ran wget on this site: http://demo.codestag.com/geeklove/ but it only returns a single index.html

wget --recursive --no-clobber --page-requisites --html-extension --convert-links --restrict-file-names=windows --domains --no-parent http://demo.codestag.com/geeklove

I tried spoofing the useragent like here : http://www.askapache.com/linux/wget-header-trick.html#Testing_Wget_Trick but still no go.

When I used the --debug option it showed that it skipped all the other pages.

share|improve this question
Tell us the exact options you passed to wget. – scai Jun 19 '13 at 7:53
@scai I have added the command – frazras Jun 19 '13 at 7:55
up vote 3 down vote accepted

Option --domains specifies a list of domains to be followed. Because you don't specify anything after this option wget downloads only those files directly specified.

If you remove this option or replace it with --domains demo.codestag.com then wget will still refuse to download more files due to the robots.txt on this server. To ignore this file you have to specify -e robots=off.

If you run into further problems it is always helpful to specify -d (--debug) to see what bothers wget.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.