Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

A few of my processes consume 100% cpu. I'm trying to figure out which scripts are causing it

I tried running strace ps -ef:

open("/proc/PID/status", O_RDONLY) = 6
read(6, "Name:\textract\nState:\tR (running)"..., 1023) = 1023
close(6) = 0
open("/proc/PID/cmdline", O_RDONLY) = 6
read(6,

So it gets stuck trying to read /proc/PID/cmdline. I tried catting that, and it got stuck again. Something is obviously screwed in the kernel; what should I try next?

Note: rebooting doesn't work -- if I shutdown manually the problem starts again. I'm using SUSE Linux Enterprise Server 11 (x86_64), Linux 2.6.27.19


Edit: ps -e produces output, and I found there are too many greps. The number of greps varies: 250, 450, and now I see around 520 greps. I traced back and found it is the result of a cron script. I still have to understand those cron scripts. Yes, top displays results. We manually shutdown the server 2 days back. System has been running from last 2 days. I see some oracle stuff running all the time. I just did the memory test, no faults detected

share|improve this question
    
What distribution and version? What kernel version? What architecture? Does ps -e produce output? What about top? Has this system been running for a long time, what is it normally running? Have you done a memory test recently? Not being able to read /proc/$pid/cmdline looks like a kernel bug or an exploit. –  Gilles Apr 9 '11 at 21:52
    
What are the processes that consume most CPU? The bunch of greps? Perhaps the hog is some badly written query to Oracle? –  vonbrand Jan 15 '13 at 16:01
add comment

2 Answers 2

Had that just yesterday. The problem was, one process was in "uninterruptible sleep" state, shown as status D in top. ls /proc/ does not return and is not abortable. ps -ef does not return and is not abortable.

If rebooting does not help you probably have a bad sector on your DVD or hard disk and the process PID is trying to read there during startup. So technically rebooting helps, but the error re-occurs automatically.

Check with top if the process is indeed in status D, then go on from there. Boot the computer without calling this process (rescue system). Then start the program stracing it and see which files it accesses. I bet one file has bad sectors.

share|improve this answer
add comment

It looks that grep is hanged and due to cron job scheduling another process will become active after certain amount of time (as written in crontab). Multiple processes will lead an unresponsive system

Try following debugging method :

  • Change crontab entry for increasing script interval (so that your hanging script does not get executed many times)
  • Record output of top for an interval
  • Traverse the process tree from top logs and then find the process at which it is hanging
  • Then traverse the code form where the same thing is being called.
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.