I have a CGI application that takes action as soon as it's loaded. Unfortunately, google's bots found it and they are ignoring the norobots, so the actions are being triggered at random times. So I want to password protect the CGI code. I don't want the cgi-bin script to handle the passwording - I want Apache to do it before the script runs.

I can't see how to do this? Apache finds cgi-bin via a ScriptAlias. There's no Directory tag to set an AuthType on. Creating a directory for the script and handling it with <Directory> doesn't run it as a script.

Is there a trick to this?

  • Um... dropping a downvote is fine, but how about some text explaining what the problem is? I'm not an apache guru. I don't know my way around apache2.conf and the comments in it don't explain how to do this. – user15001 Sep 3 '18 at 2:19
  • 2
    You can use a Location tag to do what you want. See here - serverfault.com/questions/127708/… – slm Sep 3 '18 at 2:58
  • Google respects the robots.txt convention, so you might like to check your version of this file. – meuh Sep 3 '18 at 8:10

<Location> turns out to be the way. It's just like <Directory> but you specify a url path not a filesystem path, so /cgi-bin/particularScript is the thing to use.

In my experience, norobots.txt gets ignored if Google finds your page via bookmarks in someone's Google account. It decides to crawl it regardless. I know that's how it found it because there are no public links to the page in question; it was a privately provided url and Google didn't visit it until after someone bookmarked it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.