Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I ran wget on this site: http://demo.codestag.com/geeklove/ but it only returns a single index.html

wget --recursive --no-clobber --page-requisites --html-extension --convert-links --restrict-file-names=windows --domains --no-parent http://demo.codestag.com/geeklove

I tried spoofing the useragent like here : http://www.askapache.com/linux/wget-header-trick.html#Testing_Wget_Trick but still no go.

When I used the --debug option it showed that it skipped all the other pages.

share|improve this question
    
Tell us the exact options you passed to wget. –  scai Jun 19 '13 at 7:53
    
@scai I have added the command –  frazras Jun 19 '13 at 7:55
add comment

1 Answer

up vote 3 down vote accepted

Option --domains specifies a list of domains to be followed. Because you don't specify anything after this option wget downloads only those files directly specified.

If you remove this option or replace it with --domains demo.codestag.com then wget will still refuse to download more files due to the robots.txt on this server. To ignore this file you have to specify -e robots=off.

If you run into further problems it is always helpful to specify -d (--debug) to see what bothers wget.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.