Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

GoogleBot is hitting my server hard - and even though I have set the CrawlRate in Webmaster Tools it is still hiking up the load on my server and slowing down Apache for the rest of the normal web traffic.

Is it possible to limit / rate-limit connections per second / minute using UFW based on a user agent string?

If not how can I do it for GoogleBot's IP ranges?

share|improve this question

1 Answer 1

up vote 3 down vote accepted

You cannot do this with ufw directly, but you need to add the right iptables rules to /etc/ufw/before.rules.

I suggest you to learn iptables. As a (not optimized) starting point something like

-A ufw-before-input -p tcp --syn -dport 80 -m recent --name LIMIT_BOTS --update --seconds 60 --hitcount 4 --rcheck -j DROP
-A ufw-before-input -p tcp -dport 80 -m string --algo bm --string "NotWantedUserAgent" -m recent --name LIMIT_BOTS --set ACCEPT

could work, where you of course need to replace NotWantedUserAgent with the correct one. This rules should limit the number of new connections per minute from a specific bot - I have not tested them and do not know if they really reduce the workload from a specific bot.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.