Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

There are 5 processes which can't be killed by kill -9 $PID and executing cat /proc/$PID/cmdline will hang the current session. Maybe they're zombie processes.

Executing ps -ef or htop will also hang the current session. But top and ps -e are working fine.

So it seems that there are two problems the filesystem not responding.

This is a production machine running virtual machines, so rebooting isn't an option.

The following processes ids aren't working: 16181 16765 5985 7427 7547

The parent of these processes is init

        ├─collectd(16765)─┬─{collectd}(16776)
        │                 ├─{collectd}(16777)
        │                 ├─{collectd}(16778)
        │                 ├─{collectd}(16779)
        │                 ├─{collectd}(16780)
        │                 └─{collectd}(16781)
        ├─collectd(28642)───{collectd}(28650)
        ├─collectd(29868)─┬─{collectd}(29873)
        │                 ├─{collectd}(29874)
        │                 ├─{collectd}(29875)
        │                 └─{collectd}(29876)

And one of the qemu processes not working

|-qemu-system-x86(16181)-+-{qemu-system-x86}(16232)
|                        |-{qemu-system-x86}(16238)
|                        |-{qemu-system-x86}(16803)
|                        |-{qemu-system-x86}(17990)
|                        |-{qemu-system-x86}(17991)
|                        |-{qemu-system-x86}(17992)
|                        |-{qemu-system-x86}(18062)
|                        |-{qemu-system-x86}(18066)
|                        |-{qemu-system-x86}(18072)
|                        |-{qemu-system-x86}(18073)
|                        |-{qemu-system-x86}(18074)
|                        |-{qemu-system-x86}(18078)
|                        |-{qemu-system-x86}(18079)
|                        |-{qemu-system-x86}(18086)
|                        |-{qemu-system-x86}(18088)
|                        |-{qemu-system-x86}(18092)
|                        |-{qemu-system-x86}(18107)
|                        |-{qemu-system-x86}(18108)
|                        |-{qemu-system-x86}(18111)
|                        |-{qemu-system-x86}(18113)
|                        |-{qemu-system-x86}(18114)
|                        |-{qemu-system-x86}(18119)
|                        |-{qemu-system-x86}(23147)
|                        `-{qemu-system-x86}(27051)
share|improve this question
    
Zombie processes should not cause a problem. If the number of Zombie processes are so large that exceeds the process limit on the server then it will cause a problem. –  Raza Jul 1 '13 at 5:35
    
@Salton: We can't use ps -ef and htop so we got some problems, maybe what we are seeing is not called a zombie process? –  Sam Stoelinga Jul 1 '13 at 5:47
1  
You can try to trace by using /usr/bin/strace ps -ef to see where exactly your ps -ef is hanging. –  Raza Jul 1 '13 at 6:59
1  
How did you determine these are zombies? This rather looks like hanging processes. Does ps -el work and which state are these processes in? –  Nils Jul 1 '13 at 13:41
    
In the end client still decided to reboot the machine as problems were getting worse and worse. Thanks for all the input. Learned a lot about zombies and uninteruptable processes. –  Sam Stoelinga Jul 3 '13 at 7:39

4 Answers 4

The other answers are assuming these are zombie processes. A zombie process is a process that has finished running, but is still in the process table in case the parent wants to know the exit status. These are normal, and init will automatically clean up zombie processes that get assigned to it.

Zombie processes should never cause anything to hang, so it sounds like that may not be your problem. If it's a system call or driver hanging, then the process may be in an uninterruptable state. There's a good explanation here.

share|improve this answer

You don't have zombies. cat /proc/$PID/cmdline wouldn't have any problem with a zombie. If kill -9 doesn't kill the program, it means the program is doing some uninterruptible I/O operation. That usually indicates one of three things:

  • a network filesystem that isn't responding;
  • a kernel bug;
  • a hardware bug.

Utilities such as ps may hang if they try to read some information such as the process executable path that the kernel isn't providing for one of the reasons above.

Try cat /proc/16181/syscall to see what process 16181 is doing. This may or may not work depending on how far gone your system is.

If the problem is a network filesystem, you may be able to force-unmount it, or to make it come online. If the problem is a kernel or hardware bug, what you can do will depend on the nature of the bug. Rebooting (and upgrading to a fixed kernel, or replacing the broken hardware) is strongly recommended.

share|improve this answer

You can only kill a zombie by killing its parent. A zombie process has released all its resources and is waiting for its exit status to be picked up by its parent. It becomes a zombie when the parent does not execute a wait to pick up the exit status from its child. When you kill the zombie's parent, init picks up the exit status and zombie finally dies.

share|improve this answer
    
So you want me to kill init? It's not clear from the question sorry hehe but the parent seems to be init :( I've edited the question. –  Sam Stoelinga Jul 1 '13 at 3:40
4  
No, we want you to not try to kill the zombie. You cannot kill a zombie. This FAQ is as old as Unix itself. –  tripleee Jul 1 '13 at 4:56
    
@tripleee: Yea thats what I understood. Maybe what I'm having is not a zombie. htop isn't working and cat /proc/$pid/cmdline or ls /proc/$pid/ also isn't working. Normally this doesn't happen with zombies thats why I asked here, this isn't a common problem imo. I've checked several answers which all tell you to kill the parent, which is init in my case or to reboot. –  Sam Stoelinga Jul 1 '13 at 5:48
    
"Kill the parent" is the way to reap a regular zombie. You cannot kill init. If a zombie is reparented under init, you cannot kill it. –  tripleee Jul 1 '13 at 6:08

To find zombie processes on Linux:

# ps axo stat,ppid,pid,comm | grep -w defunct

Z 555 10242 Damn-Zombie < defunct >

First, you can try sending SIGCHLD signal to the zombie’s parent process using the kill command. Note that the above command gives you PPID (PID of parent process) of each zombie. In our example, PPID of the zombie is 250.

# sudo kill -s SIGCHLD 555

If a zombie process still does not go away, you can kill the parent process (e.g., 250) of the zombie.

# sudo kill -9 555

Once its parent process gets killed, the zombie will be adopted by the init process, which is a parent of all processes in Linux. The init process periodically calls wait() to reap any zombie process.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.