Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
I have a list of bots to block, so I was thinking that fail2ban could be a solution until I realized that mod_security would be more efficient in this kind of tasks.
The number of bots is huge, so the file of configuration will contain a long list.
My question is about performance (memory, processor, disk ..etc):
Is having a huge list of bots to block will affect the performance of apache in a site with huge traffic ?
Performance should not be problem; check @rbl feature of ModSecurity in combination with local DNS caching which will enable you to keep track of offenders in DNS and utilize that DNS zone as reference when deciding to allow or block some connection.
Instead of managing a long list of bots, you can filter bots out based on a few keywords present in the user-agent string and only let certain bots through.
I use this to block all spiders/bots i don't want crawling my site. Then out of those i only let those through i am interested in indexing the website. All others get blocked by the firewall.
So far this performs quite well for me.
sudo nohup tail -f /var/log/apache2/access.log | awk -F' - |\"' 'tolower($7) ~ /bot|crawler|spider/ && tolower($7) !~ /google|yahoo|bing|msn|ask|aol|duckduckgo/{system(" sudo ufw deny from "$1" to any comment \""$7"\"")}' > /dev/null &
