Get CPU usage and run a command if it is higher than 80%

Question

My VPS was hacked several times, hackers put a CPU miner. My hosting provider shutdowns VPS if miner is detected and I did not reactin the next 12 hours. But they can send me notice at 21.00 PM on Saturday :) , and shutdown server at 9.00 AM on Sunday.

So I want to monitor CPU usage and block folder where miner is always revelead from writing.

I'm not very familiar with Linux, so please suggest with such script

1. Check CPU usage, if it is higher than 80% (as example) Do something.
2. In my case - delete all from install folder and make it read only.

Actually I've no idea how to implement item №1.

Answer 1

I had a similar issue and had this short bash script already done. It is calculating the load average for the last 15 minutes, if you want a different timeframe, it shold be change (to check the load avg for last 5 min, change the awk to print $1).
This will tell you the relative usage of the CPUs :

#!/bin/bash
cores=$(nproc) 
load=$(awk '{print $3}'< /proc/loadavg)
echo | awk -v c="${cores}" -v l="${load}" '{print "relative load is " l*100/c "%"}'

Should run on Ubuntu and Centos.

To get to the point where you check if load is above 80% and 'do something' you should add to this script :

usage=$(echo | awk -v c="${cores}" -v l="${load}" '{print l*100/c}' | awk -F. '{print $1}')
if [[ ${usage} -ge 80 ]]; then
    echo "delete all from install folder and make it read only"
fi

Answer 2

Check CPU usage, if it is higher than 80% (as example) Do something.

Modern CPUs have multiple cores and often each core supports multiple threads. Moreover, cores have often variable clock speed. It is then not that simple to define how to compute a CPU load. Not to mention a single vCPU can only be either 100% idle or 100% busy at any given time. There is no such thing as instantaneous 80% busy CPU.

What you can do is get the average load for each vCPU during a period of time (mpstat provides that) or the average for all vCPUs combined (vmstat).

Even fully CPU bound, if the hostile CPU consumer is single-threaded, it might not blatantly show up in the latter case because other vCPUs might be idle.

If multi threaded and CPU bound, it will be detected by both commands, but you have to make sure it is not a legit application or daemon which is loading your machine.

Another more useful metric is derived from the CPU contention, i.e. measuring how many threads are using and competing for the vCPU resources. This is what the load average is designed to show. Unfortunately, on Linux, the load average is considering a thread uninterruptible state to be CPU load while in fact, the CPU is idle and free to do other tasks so you should pay attention to that factor and identify potential cases where the load average is high but the actual contention is low.

Finally, there might have situations where the run queue is very high but for a very limited period of time. If the load average calculation which use sampling to get the run queue value happen to pick the number at this very peak moment, the load average value will be strongly biased for several minutes or dozens of minutes.