My server is often almost DOS'ed by crawlers. I have not yet managed to figure out what exactly is responsible for the high load. CPU, memory and disk IO are not at the limit (could be network traffic). My question is, if anyone has an idea how to reserve capacity in the system to run a shell session (bash over ssh). Can this be done using cgroups? Right now load values easily go up to 60 and that make it hard to enter any command.
Something more basic: start
IF the problem is due to DOS throttling the network, then you have to use network-shaping rules outside the box (ie, on the switch).