9

A few of my processes consume 100% cpu. I'm trying to figure out which scripts are causing it

I tried running strace ps -ef:

open("/proc/PID/status", O_RDONLY) = 6
read(6, "Name:\textract\nState:\tR (running)"..., 1023) = 1023
close(6) = 0
open("/proc/PID/cmdline", O_RDONLY) = 6
read(6,

So it gets stuck trying to read /proc/PID/cmdline. I tried catting that, and it got stuck again. Something is obviously screwed in the kernel; what should I try next?

Note: rebooting doesn't work -- if I shutdown manually the problem starts again. I'm using SUSE Linux Enterprise Server 11 (x86_64), Linux 2.6.27.19

Edit: ps -e produces output, and I found there are too many greps. The number of greps varies: 250, 450, and now I see around 520 greps. I traced back and found it is the result of a cron script. I still have to understand those cron scripts. Yes, top displays results. We manually shutdown the server 2 days back. System has been running from last 2 days. I see some oracle stuff running all the time. I just did the memory test, no faults detected

Improve this question
2
  • What distribution and version? What kernel version? What architecture? Does ps -e produce output? What about top? Has this system been running for a long time, what is it normally running? Have you done a memory test recently? Not being able to read /proc/$pid/cmdline looks like a kernel bug or an exploit.
    – Gilles 'SO- stop being evil'
    Apr 9, 2011 at 21:52
  • What are the processes that consume most CPU? The bunch of greps? Perhaps the hog is some badly written query to Oracle?
    – vonbrand
    Jan 15, 2013 at 16:01

2 Answers 2

Reset to default
2

Had that just yesterday. The problem was, one process was in "uninterruptible sleep" state, shown as status D in top. ls /proc/ does not return and is not abortable. ps -ef does not return and is not abortable.

If rebooting does not help you probably have a bad sector on your DVD or hard disk and the process PID is trying to read there during startup. So technically rebooting helps, but the error re-occurs automatically.

Check with top if the process is indeed in status D, then go on from there. Boot the computer without calling this process (rescue system). Then start the program stracing it and see which files it accesses. I bet one file has bad sectors.

Improve this answer
1
  • 1
    This is exactly what was happening to us at work today. We noticed a process stuck in uninterruptible sleep D, adjusted logging to see more details, reran, and then ps -ef and pgrep both started hanging. We noticed it inside podman container, but eventually noticed, even the host OS could no longer ps -ef without hanging.
    – Ajax
    May 25, 2023 at 1:32
0

It looks that grep is hanged and due to cron job scheduling another process will become active after certain amount of time (as written in crontab). Multiple processes will lead an unresponsive system

Try following debugging method :

  • Change crontab entry for increasing script interval (so that your hanging script does not get executed many times)
  • Record output of top for an interval
  • Traverse the process tree from top logs and then find the process at which it is hanging
  • Then traverse the code form where the same thing is being called.
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .