Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

My website is being DoS'ed by Google webspiders. Google is welcome to index my site, but sometimes it is querying a tagcloud on my site faster than my webserver can produce the results, making my webserver run out of resources.

How can I limit access to my webserver in such a way that normal visitors are not affected?

robots.txt is no option because it would block the whole site from being indexed.

iptables -m recent is tricky, because some pages have a lot of images or other data files and 'recent' triggers on those too (typically my RSS aggregator, loading images and feeds).

iptables -m limit has the same disadvantage and on top of that, I wasn't able to be selective per IP source address.

How can I limit visitors that cause my server load to rise too high?

I am running apache2 on Ubuntu server in a VirtualBox VM.

share|improve this question
1  
You can use Webmaster tools to lower the crawl rate. See support.google.com/webmasters/bin/… –  derobert Oct 8 '12 at 14:34
add comment

1 Answer

up vote 4 down vote accepted

Try the mod_qos Apache module. The current version has the following control mechanisms.

  • The maximum number of concurrent requests to a location/resource (URL) or virtual host.
  • Limitation of the bandwidth such as the maximum allowed number of requests per second to an URL or the maximum/minimum of downloaded kbytes per second.
  • Limits the number of request events per second (special request conditions). It can also "detect" very important persons (VIP) which may access the web server without or with fewer restrictions.
  • Generic request line and header filter to deny unauthorized operations.
  • Request body data limitation and filtering (requires mod_parp).
  • Limitations on the TCP connection level, e.g., the maximum number of allowed connections from a single IP source address or dynamic keep-alive control.
  • Prefers known IP addresses when server runs out of free TCP connections.

This sample conditional rule from the documentation should get you going in the right direction.

# set the conditional variable to spider if detecting a
# "slurp" or "googlebot" search engine:
BrowserMatch             "slurp"                  QS_Cond=spider
BrowserMatch             "googlebot"              QS_Cond=spider

# limits the number of concurrent requests to two applications
# (/app/b and /app/c) to 300 but does not allow access by a "spider"
# if the number of concurrent requests exceeds the limit of 10:
QS_LocRequestLimitMatch       "^(/app/b/|/app/c/).*$"  300
QS_CondLocRequestLimitMatch   "^(/app/b/|/app/c/).*$"  10   spider
share|improve this answer
    
Looks a bit more complicated than I'd hoped for on a friday evening enjoying Belgian beer ... Have to look at it tomorrow. Looks promising with the browsermatch and things. Coming to think of it ... would be nice if it automatically any host / useragent that requests robot.txt :o) Thnx Uther –  jippie Apr 27 '12 at 19:58
    
Enjoy that beer! –  uther Apr 27 '12 at 20:00
    
OK, it looks pretty simple. Implemented it to see if my webserver wil be able to keep up now. –  jippie Apr 29 '12 at 12:57
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.