I have an embedded system that was going non-responsive after running for several hours. After investigating I found that the system kept growing a list of <defunct> processes. Running ps axl gives a long list a snippet of which is as follows:

1     0  6421     1  20   0      0     0 exit   Zs   ?          0:00 [timeout] <defunct>
1     0  6429     1  20   0      0     0 exit   Zs   ?          0:00 [timeout] <defunct>
1     0  6476     1  20   0      0     0 exit   Zs   ?          0:00 [timeout] <defunct>
1     0  6497     1  20   0      0     0 exit   Zs   ?          0:00 [timeout] <defunct>

I am not able to kill these processes. Increase in these processes eat up all the RAM thus making system unresponsive. The other issue is that the processes that are run at boot time by init scripts also go <defunct> when I try to kill them. I am not able to find a clue that why these zombie processes are not killed by init and eating up RAM.

  • I remember that defunct processes or zombies cannot be klled, by definition. Ofcourse a reboot will clean the list. I remember that zombies do not use RAM except the process handler that system is using to manage the processes. Maybe you should try to gather all messages generated in order to find the root cause: dmsg, /var/log/* files etc. – Jay jargot Mar 26 '16 at 10:18
  • The processes are not eating up RAM. See those columns with the zeroes? That's how much RAM the processes are using. – Gilles Mar 26 '16 at 15:07
  • What program generates these processes? It should either set SIGCHLD to SIG_IGN or call a wait() function. – ott-- Mar 26 '16 at 15:46
  • There is a script that runs these processes. But even if i run same command from command-line, the process goes <defunct>. E.g. running 'timeout 5 ping c1 google.com' makes 'timeout' process to go <defunct> – arshan Mar 28 '16 at 4:43

Defunct processes do not eat any significant ram, just a handful of bytes to store their pid, return status and resource usage statistics.

However, defunct processes parented by init should be quickly reaped by the latter on a properly running Unix/Linux OS.

There is then a serious bug with your system which might also explain the memory leak you are observing.

  • I was guessing the same thing but I was confused as there are multiple systems running same apps/scripts, but only a few show this problem. – arshan Mar 28 '16 at 4:39

As others I doubt those defunct processes use any significant anout of RAM, you can also see in the ps output you have given, that there are zeroes in the columns detailing memory usage.

Defnct (or zombie - that's where the Z comes from) processes can't be killed, the just linger in the process table until their parent reads their exit status - so you should fix the parent.

  • Right! as top command also indicates zombies are using no memory. But even if I run 'timeout 5 ping -c 2 google.com' from command-line, the timeout goes <defunct> after ping is successfully terminated. – arshan Mar 26 '16 at 10:36
  • So enabling cron that runs the scripts (which execute timeout command) starts increasing <defunct> list as well as the free RAM starts going down. – arshan Mar 26 '16 at 10:38

Find Zombies using #ps auxwww | grep -w Z | grep defunct | grep -v grep

Check the parent process ids of these zombies and Kill it.

  • Parent PID was 1 (i.e. init) hence could not be killed. – arshan Apr 2 '16 at 7:15

The root cause of the problem was the faulty hardware. ttyS1 port of the system was receiving thousands of interrupts per second, which in turn resulted in high CPU usage by kworker. This caused the creation of heap of <defucnt> processes and were not cleaned by init. This could be bug of init as suggested by @jilliagre.

Disabling getty on ttyS1 solved the problem.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.