15

There are 5 processes which can't be killed by kill -9 $PID and executing cat /proc/$PID/cmdline will hang the current session. Maybe they're zombie processes.

Executing ps -ef or htop will also hang the current session. But top and ps -e are working fine.

So it seems that there are two problems the filesystem not responding.

This is a production machine running virtual machines, so rebooting isn't an option.

The following processes ids aren't working: 16181 16765 5985 7427 7547

The parent of these processes is init

        ├─collectd(16765)─┬─{collectd}(16776)
        │                 ├─{collectd}(16777)
        │                 ├─{collectd}(16778)
        │                 ├─{collectd}(16779)
        │                 ├─{collectd}(16780)
        │                 └─{collectd}(16781)
        ├─collectd(28642)───{collectd}(28650)
        ├─collectd(29868)─┬─{collectd}(29873)
        │                 ├─{collectd}(29874)
        │                 ├─{collectd}(29875)
        │                 └─{collectd}(29876)

And one of the qemu processes not working

|-qemu-system-x86(16181)-+-{qemu-system-x86}(16232)
|                        |-{qemu-system-x86}(16238)
|                        |-{qemu-system-x86}(16803)
|                        |-{qemu-system-x86}(17990)
|                        |-{qemu-system-x86}(17991)
|                        |-{qemu-system-x86}(17992)
|                        |-{qemu-system-x86}(18062)
|                        |-{qemu-system-x86}(18066)
|                        |-{qemu-system-x86}(18072)
|                        |-{qemu-system-x86}(18073)
|                        |-{qemu-system-x86}(18074)
|                        |-{qemu-system-x86}(18078)
|                        |-{qemu-system-x86}(18079)
|                        |-{qemu-system-x86}(18086)
|                        |-{qemu-system-x86}(18088)
|                        |-{qemu-system-x86}(18092)
|                        |-{qemu-system-x86}(18107)
|                        |-{qemu-system-x86}(18108)
|                        |-{qemu-system-x86}(18111)
|                        |-{qemu-system-x86}(18113)
|                        |-{qemu-system-x86}(18114)
|                        |-{qemu-system-x86}(18119)
|                        |-{qemu-system-x86}(23147)
|                        `-{qemu-system-x86}(27051)
Improve this question
5
  • 1
    Zombie processes should not cause a problem. If the number of Zombie processes are so large that exceeds the process limit on the server then it will cause a problem.
    – Raza
    Commented Jul 1, 2013 at 5:35
  • @Salton: We can't use ps -ef and htop so we got some problems, maybe what we are seeing is not called a zombie process?
    – Sam Stoelinga
    Commented Jul 1, 2013 at 5:47
  • 1
    You can try to trace by using /usr/bin/strace ps -ef to see where exactly your ps -ef is hanging.
    – Raza
    Commented Jul 1, 2013 at 6:59
  • 2
    How did you determine these are zombies? This rather looks like hanging processes. Does ps -el work and which state are these processes in?
    – Nils
    Commented Jul 1, 2013 at 13:41
  • In the end client still decided to reboot the machine as problems were getting worse and worse. Thanks for all the input. Learned a lot about zombies and uninteruptable processes.
    – Sam Stoelinga
    Commented Jul 3, 2013 at 7:39

4 Answers 4

Reset to default
23

You don't have zombies. cat /proc/$PID/cmdline wouldn't have any problem with a zombie. If kill -9 doesn't kill the program, it means the program is doing some uninterruptible I/O operation. That usually indicates one of three things:

  • a network filesystem that isn't responding;
  • a kernel bug;
  • a hardware bug.

Utilities such as ps may hang if they try to read some information such as the process executable path that the kernel isn't providing for one of the reasons above.

Try cat /proc/16181/syscall to see what process 16181 is doing. This may or may not work depending on how far gone your system is.

If the problem is a network filesystem, you may be able to force-unmount it, or to make it come online. If the problem is a kernel or hardware bug, what you can do will depend on the nature of the bug. Rebooting (and upgrading to a fixed kernel, or replacing the broken hardware) is strongly recommended.

Improve this answer
1
  • Cat never responds. I don't think this is a bug. i think it's a "feature".
    – Owl
    Commented Mar 21, 2019 at 17:16
7

The other answers are assuming these are zombie processes. A zombie process is a process that has finished running, but is still in the process table in case the parent wants to know the exit status. These are normal, and init will automatically clean up zombie processes that get assigned to it.

Zombie processes should never cause anything to hang, so it sounds like that may not be your problem. If it's a system call or driver hanging, then the process may be in an uninterruptable state. There's a good explanation here.

Improve this answer
1
  • 1
    Too many zombie processes can prevent fork from succeeding (when hard nrpoc is reached) because they still occupy space in the process table.
    – dhchdhd
    Commented Apr 28, 2017 at 1:56
4

To find zombie processes on Linux:

$ ps axo stat,ppid,pid,comm | grep -w defunct

Z 555 10242 Damn-Zombie < defunct >

First, you can try sending SIGCHLD signal to the zombie’s parent process using the kill command. Note that the above command gives you PPID (PID of parent process) of each zombie. In our example, PPID of the zombie is 555.

$ sudo kill -s SIGCHLD 555

If a zombie process still does not go away, you can kill the parent process (e.g., 555) of the zombie.

$ sudo kill -9 555

Once its parent process gets killed, the zombie will be adopted by the init process, which is a parent of all processes in Linux. The init process periodically calls wait() to reap any zombie process.

Improve this answer
1
  • 1
    This is the real answer. Killing the parent worked, thanks.
    – Andrew
    Commented Apr 30, 2019 at 3:26
1

You can only kill a zombie by killing its parent. A zombie process has released all its resources and is waiting for its exit status to be picked up by its parent. It becomes a zombie when the parent does not execute a wait to pick up the exit status from its child. When you kill the zombie's parent, init picks up the exit status and zombie finally dies.

Improve this answer
4
  • So you want me to kill init? It's not clear from the question sorry hehe but the parent seems to be init :( I've edited the question.
    – Sam Stoelinga
    Commented Jul 1, 2013 at 3:40
  • 4
    No, we want you to not try to kill the zombie. You cannot kill a zombie. This FAQ is as old as Unix itself.
    – tripleee
    Commented Jul 1, 2013 at 4:56
  • @tripleee: Yea thats what I understood. Maybe what I'm having is not a zombie. htop isn't working and cat /proc/$pid/cmdline or ls /proc/$pid/ also isn't working. Normally this doesn't happen with zombies thats why I asked here, this isn't a common problem imo. I've checked several answers which all tell you to kill the parent, which is init in my case or to reboot.
    – Sam Stoelinga
    Commented Jul 1, 2013 at 5:48
  • 1
    "Kill the parent" is the way to reap a regular zombie. You cannot kill init. If a zombie is reparented under init, you cannot kill it.
    – tripleee
    Commented Jul 1, 2013 at 6:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .