3

I have an embedded system that was going non-responsive after running for several hours. After investigating I found that the system kept growing a list of <defunct> processes. Running ps axl gives a long list a snippet of which is as follows:

1     0  6421     1  20   0      0     0 exit   Zs   ?          0:00 [timeout] <defunct>
1     0  6429     1  20   0      0     0 exit   Zs   ?          0:00 [timeout] <defunct>
1     0  6476     1  20   0      0     0 exit   Zs   ?          0:00 [timeout] <defunct>
1     0  6497     1  20   0      0     0 exit   Zs   ?          0:00 [timeout] <defunct>

I am not able to kill these processes. Increase in these processes eat up all the RAM thus making system unresponsive. The other issue is that the processes that are run at boot time by init scripts also go <defunct> when I try to kill them. I am not able to find a clue that why these zombie processes are not killed by init and eating up RAM.

5
  • I remember that defunct processes or zombies cannot be klled, by definition. Ofcourse a reboot will clean the list. I remember that zombies do not use RAM except the process handler that system is using to manage the processes. Maybe you should try to gather all messages generated in order to find the root cause: dmsg, /var/log/* files etc.
    – Jay jargot
    Mar 26, 2016 at 10:18
  • The processes are not eating up RAM. See those columns with the zeroes? That's how much RAM the processes are using. Mar 26, 2016 at 15:07
  • What program generates these processes? It should either set SIGCHLD to SIG_IGN or call a wait() function.
    – ott--
    Mar 26, 2016 at 15:46
  • There is a script that runs these processes. But even if i run same command from command-line, the process goes <defunct>. E.g. running 'timeout 5 ping c1 google.com' makes 'timeout' process to go <defunct>
    – arshan
    Mar 28, 2016 at 4:43
  • This is how I remain immortal. I remain in <defunct> state.
    – Andrew
    Aug 28, 2019 at 3:00

4 Answers 4

3

Defunct processes do not eat any significant ram, just a handful of bytes to store their pid, return status and resource usage statistics.

However, defunct processes parented by init should be quickly reaped by the latter on a properly running Unix/Linux OS.

There is then a serious bug with your system which might also explain the memory leak you are observing.

1
  • I was guessing the same thing but I was confused as there are multiple systems running same apps/scripts, but only a few show this problem.
    – arshan
    Mar 28, 2016 at 4:39
1

As others I doubt those defunct processes use any significant anout of RAM, you can also see in the ps output you have given, that there are zeroes in the columns detailing memory usage.

Defnct (or zombie - that's where the Z comes from) processes can't be killed, the just linger in the process table until their parent reads their exit status - so you should fix the parent.

2
  • Right! as top command also indicates zombies are using no memory. But even if I run 'timeout 5 ping -c 2 google.com' from command-line, the timeout goes <defunct> after ping is successfully terminated.
    – arshan
    Mar 26, 2016 at 10:36
  • So enabling cron that runs the scripts (which execute timeout command) starts increasing <defunct> list as well as the free RAM starts going down.
    – arshan
    Mar 26, 2016 at 10:38
0

Find Zombies using #ps auxwww | grep -w Z | grep defunct | grep -v grep

Check the parent process ids of these zombies and Kill it.

1
  • Parent PID was 1 (i.e. init) hence could not be killed.
    – arshan
    Apr 2, 2016 at 7:15
0

The root cause of the problem was the faulty hardware. ttyS1 port of the system was receiving thousands of interrupts per second, which in turn resulted in high CPU usage by kworker. This caused the creation of heap of <defucnt> processes and were not cleaned by init. This could be bug of init as suggested by @jilliagre.

Disabling getty on ttyS1 solved the problem.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .