Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

As an example - http://aok.heavengames.com/cgi-bin/aokcgi/display.cgi?action=t&fn=22.

I found a way to get through the robots.txt restrictions, but even then, it just downloads a binary file that's unreadable by anything.

share|improve this question
wget --user-agent=Mozilla \
  "http://aok.heavengames.com/cgi-bin/aokcgi/display.cgi?action=t&fn=22"
share|improve this answer

From the wget manual on gnu.org

If you know what you are doing and really really wish to turn off the robot exclusion, set the robots variable to ‘off’ in your .wgetrc. You can achieve the same effect from the command line using the -e switch, e.g. ‘wget -e robots=off url...’.

share|improve this answer
    
Actually after actually trying this now, Ulrich Dangel's solution works perfectly because the website is 403 on unknown user agent's it seems. – Anthony Hatzopoulos Jul 20 '12 at 2:31

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.