Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have created a script that kills processes if CPU and/or memory usage hits 80%. It creates a list of killed processes when this happens. What can I do to improve it?

while [ 1 ];
do 
echo
echo checking for run-away process ...

CPU_USAGE=$(uptime | cut -d"," -f4 | cut -d":" -f2 | cut -d" " -f2 | sed -e "s/\.//g")
CPU_USAGE_THRESHOLD=800
PROCESS=$(ps aux r)
TOPPROCESS=$(ps -eo pid -eo pcpu -eo command | sort -k 2 -r | grep -v PID | head -n 1)

if [ $CPU_USAGE -gt $CPU_USAGE_THRESHOLD] ; then
  kill -9 $(ps -eo pid | sort -k 1 -r | grep -v PID | head -n 1) #original
  kill -9 $(ps -eo pcpu | sort -k 1 -r | grep -v %CPU | head -n 1)
  kill -9 $TOPPROCESS
  echo system overloading!
  echo Top-most process killed $TOPPROCESS
      echo CPU USAGE is at $CPU_LOAD

else
    fi
    exit 0
    sleep 1;
    done
share|improve this question
2  
Have you tried running the script? while [ 1 ] makes me wonder how much CPU this script alone is going to consume. Also, 3 calls to kill -9 in a script running constantly? This gives me the chills ... –  rahmu Mar 4 '13 at 7:55
1  
Nice avatar anyway, @rahmu he got a sleep 1 in the loop –  warl0ck Mar 4 '13 at 8:16
    
First monday of the month and my PC is checking a (slow'ish) RAID6 set. CPU load easily peaks above 8 because it is constantly waiting for disk IO from this RAID set. Nothing wrong, system is still very responsive. Your script would kill my firefox which uses only 3.6% of the available 400%. Just saying you may be hunting for ghosts with this script. BTW: your system will not get damaged because of high load and when memory runs out, the kernel will do a semi-educated guess on which process(es) to kill. –  jippie Mar 4 '13 at 8:28
    
Then kill process on based load will be fine or not ???? –  K.K Patel Mar 4 '13 at 9:10
    
In my use case the killing is not desired. –  jippie Mar 4 '13 at 18:48

3 Answers 3

up vote 3 down vote accepted

I'm guessing the problem you want to solve is that you have some process running on your box which sometimes misbehaves, and sits forever pegging a core.

The first thing you want to do is to attempt to fix the program that goes crazy. That is by far the best solution. I'm going to assume that isn't possible, or you need a quick kluge to keep your box running until its fixed.

You, at minimum, want to limit your script to only hit the one program you're concerned about. It'd be best if permissions limited your script like this (e.g., your script runs as user X, the only other thing running as X is the program).

Even better would be to use something like ulimit -t to limit the amount of total CPU time that the program can use. Similarly, if it consumes all memory, check ulimit -v. The kernel enforces these limits; see the bash manpage (it's a shell built-in) and the setrlimit(2) manpage for details.

If the problem isn't a process running amok, but is instead just too many processes running, then implement some form of locking to prevent more than X from running (or—this should be getting familiar—ulimit -u). You may also consider changing the scheduler priority of those processes (using nice or renice), or for even more drastic, using sched_setscheduler to change the policy to SCHED_IDLE.

If you need even more control, take a look a control groups (cgroups). Depending on the kernel you're running, you can actually limit the amount of CPU time, memory, I/O, etc. that a whole group of processes together consume. Control groups are quite flexible; they can likely do whatever you're trying to do, without any fragile kluges. The Arch Linux Wiki has an intro to cgroups that's worth reading.

share|improve this answer

Killing off processes which are using most CPU/memory is asking for trouble: Just look at what they are right now on your machine (here currently firefox, systemd (init), Xorg, gnome-terminal, a set of kernel threads, xemacs; none of which is dispensable). Look at how to tweak Linux' OOM-killer, for example here.

Also note that "memory used by the process" is a nebulous concept, as there are shared libraries, executables are shared, and even parts of data areas. One can come up with some number by charging each user with a fraction of the used space, but even adding that up really doesn't give "memory used" (even less "memory freed if the process goes away", the parts shared stay behind).

share|improve this answer

Issues:

  • When sorting numeric fields you probably want to use the -n option: sort -nrk 2. Otherwise a line with a %CPU value of 5.0 will end up higher than one with a value of 12.0.
  • Depending on your ps implementation you might want to use the --no-headers option to get rid of the grep -v. That prevents you from discarding commands that contains PID.
  • I guess instead of echo CPU USAGE is at $CPU_LOAD, you meant echo CPU USAGE is at $CPU_USAGE.
  • I guess you forgot to remove the exit 0 that you inserted during debugging(?).

Style:

  • You might want to move the CPU_USAGE_THRESHOLD=800 line to the beginning of the file, as this the most informative thing and is most likely to get changed even after your script is stable.
  • You are repeating the -e option: ps -eo pid -eo pcpu -eo command is the same as ps -eo pid -o pcpu -o command (as is ps -eo pid,pcpu,command).
  • There is an empty else clause. That always looks as if it should be handled, but was not for some unknown reason.
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.