2

I ran wget on this site: http://demo.codestag.com/geeklove/ but it only returns a single index.html

wget --recursive --no-clobber --page-requisites --html-extension --convert-links --restrict-file-names=windows --domains --no-parent http://demo.codestag.com/geeklove

I tried spoofing the useragent like here : http://www.askapache.com/linux/wget-header-trick.html#Testing_Wget_Trick but still no go.

When I used the --debug option it showed that it skipped all the other pages.

2
  • Tell us the exact options you passed to wget.
    – scai
    Commented Jun 19, 2013 at 7:53
  • @scai I have added the command
    – frazras
    Commented Jun 19, 2013 at 7:55

1 Answer 1

4

Option --domains specifies a list of domains to be followed. Because you don't specify anything after this option wget downloads only those files directly specified.

If you remove this option or replace it with --domains demo.codestag.com then wget will still refuse to download more files due to the robots.txt on this server. To ignore this file you have to specify -e robots=off.

If you run into further problems it is always helpful to specify -d (--debug) to see what bothers wget.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .