1

when running top it shows a couple of processes got nearly 100% cpu usage of the command 'perl'

    PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 21287  simon      15   0  4141  231 5891 S  90.3  0.0  14:27.50 perl
 22545 albert      15   0 12744 1172 8321 S  99.0  0.0  20:00.58 perl
        ...
        ...

lsof -p <pid> to estimate if it's a malicious process (usually somewhere down the pipe line it's trying to connect to a foreign ip address using a local port > 50000)

kill -9 <pid> to kill those malicious process individually

then repeat the above

ideally I would like to automatically kill processes of:

  • users shown above
  • have been runing more than 15 minutes
  • more than 90% cpu usage.
  • with outbound tcp connection (if possible)

does anyone happen to have a handy script doing so? thanks!

1

Actually I don't have a script handy but killing these malicious processes won't do anything to solve you problem that you have these in the first place... I guess you are a webhost and these are miners that running on your VPS?

What you could do is instead of killing them to "starve them to death" using cgroups.

You'll have CPU and Network accounting there and a tool called cgclassify. You can build flexible hierarchies. There is some good documentation for CentOS/RHEL here.

I'm not an expert on the details but I'll think it's possible in this way: You create cgroups for CPU/Memory/Network and classify the processes according to used shares using cgclassify or the mentionend daemon. That means the daemons take care of of supervising ressource usage and you can move e.g. processes that use the network and a certain amount of CPU shares into a cgroup that severely limits CPU and Network access or outright kills the process (e.g. due to low memory).

Here is a presention with some details.

The problem with killing these processes every 15 minutes is that these are likely be respawned instantly and that they hog your complete resources in these 15 minutes.

With cgroups you can classify certain safe binaries (e.g. PHP, Apache) and limit all other processes... so using CPU/Network will not impact the server and it's soo slow that it's not worthwhile for the users to run these tasks.. It's also possible to shape the CPU shares in such a way that the process will get slower the more CPU it's using. The same for Network bandwidth.. you can limit e.g. all processes for each user to only use 1Mbit for outgoing network connections or even less.

You can also just use the accounting data from the cgroups to kill the processes.

Doing this "on the fly" looks like another problem but if the problem is persistent look into using and modifying ulatencyd or similiar tools to your needs. With good rules this should take care of problematic processes faster than 15 Minutes and with some persistance.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.