How do I get wget to download a cgi file behind robots.txt?

Question

As an example - http://aok.heavengames.com/cgi-bin/aokcgi/display.cgi?action=t&fn=22.

I found a way to get through the robots.txt restrictions, but even then, it just downloads a binary file that's unreadable by anything.

Answer 1

wget --user-agent=Mozilla \
  "http://aok.heavengames.com/cgi-bin/aokcgi/display.cgi?action=t&fn=22"

Answer 2

From the wget manual on gnu.org

If you know what you are doing and really really wish to turn off the robot exclusion, set the robots variable to ‘off’ in your .wgetrc. You can achieve the same effect from the command line using the -e switch, e.g. ‘wget -e robots=off url...’.