452

I have a process I can't kill with kill -9 <pid>. What's the problem in such a case, especially since I am the owner of that process. I thought nothing could evade that kill option.

14 Answers 14

544

kill -9 (SIGKILL) always works, provided you have the permission to kill the process. Basically either the process must be started by you and not be setuid or setgid, or you must be root. There is one exception: even root cannot send a fatal signal to PID 1 (the init process).

However kill -9 is not guaranteed to work immediately. All signals, including SIGKILL, are delivered asynchronously: the kernel may take its time to deliver them. Usually, delivering a signal takes at most a few microseconds, just the time it takes for the target to get a time slice. However, if the target has blocked the signal, the signal will be queued until the target unblocks it.

Normally, processes cannot block SIGKILL. But kernel code can, and processes execute kernel code when they call system calls. Kernel code blocks all signals when interrupting the system call would result in a badly formed data structure somewhere in the kernel, or more generally in some kernel invariant being violated. So if (due to a bug or misdesign) a system call blocks indefinitely, there may effectively be no way to kill the process. (But the process will be killed if it ever completes the system call.)

A process blocked in a system call is in uninterruptible sleep. The ps or top command will (on most unices) show it in state D (originally for “disk”, I think).

A classical case of long uninterruptible sleep is processes accessing files over NFS when the server is not responding; modern implementations tend not to impose uninterruptible sleep (e.g. under Linux, the intr mount option allows a signal to interrupt NFS file accesses).

You may sometimes see entries marked Z (or H under Linux, I don't know what the distinction is) in the ps or top output. These are technically not processes, they are zombie processes, which are nothing more than an entry in the process table, kept around so that the parent process can be notified of the death of its child. They will go away when the parent process pays attention (or dies).

  • 86
    Yor reply looks self contradicting. You start telling SIGKILL always works but end citing the uninterruptible sleep case, where SIGKILL might never work outside shutting down the kernel. There are also two cases where SIGKILL doesn't work. With zombies obviously as you can't kill already dead processes and with init, which by design is ignoring SIGKILL signals. – jlliagre Jan 11 '11 at 12:27
  • 40
    @jlliagre: Killing a zombie doesn't make sense, it's not alive to begin with. And killing a process in interruptible sleep does work, it's just (as with other signals) asynchronous. I've tried to clarify this in my edit. – Gilles Jan 11 '11 at 20:07
  • 3
    I wrote too killing a zombie doesn't make sense but that doesn't prevent many people to try it and complain. Killing a process in interruptible sleep indeed works by design, but I was talking about killing a process in uninterruptible sleep which can fail if the system call never wake up. – jlliagre Jan 11 '11 at 21:39
  • 10
    man 5 nfs: "The intr/nointr mount option is deprecated after kernel 2.6.25. Only SIGKILL can interrupt a pending NFS operation on these kernels, and if specified, this mount option is ignored to provide backwards compatibility with older kernels." – Martin Schröder Aug 7 '12 at 20:05
  • 4
    @imz--IvanZakharyaschev Not that I know of (but I might not know). With sshfs, as a last resort, you can kill the sshfs process (and likewise with any other FUSE filesystem: you can always force-unmount this way). – Gilles Mar 28 '13 at 19:42
98

Sometime process exists and cannot be killed due to:

  • being zombie. I.e. process which parent did not read the exit status. Such process does not consume any resources except PID entry. In top it is signaled Z
  • erroneous uninterruptible sleep. It should not happen but with a combination of buggy kernel code and/or buggy hardware it sometime does. The only method is to reboot or wait. In top it is signaled by D.
  • 2
    Zombie doesn't consume resource ? – Luc M Jan 11 '11 at 4:17
  • 7
    @Luc M: AFAIK no (at least on Linux) - with exception of the entry in process table (i.e. PID along with such information as owner, exit status etc.). It is just process which wait acknowledgement from partent that it terminated. – Maciej Piechotka Jan 11 '11 at 5:16
  • 18
    @xenoterracide: Eventually yes but if parent process still lives (for example it is gnome-session or something which fullfill similar role) you still may have zombies. Technically it is parent job to clean up but if zombie is orphaned init cleans after it (terminology is the reason why the unix classes are done with closed doors - anyone hearing about orphans, zombies and killing in one sentence may have got wrong impressions). – Maciej Piechotka Jan 11 '11 at 10:13
  • 5
    "...only method is to reboot or wait. " Wait how long? Five months have gone by and my zombies are still there. – DarenW Apr 14 '15 at 4:10
  • 3
    @DarenW until the parent acknowledges the death of children. For details please ask the author of the program. – Maciej Piechotka Jan 16 '16 at 5:35
32

It sounds like you might have a zombie process. This is harmless: the only resource a zombie process consumes is an entry in the process table. It will go away when the parent process dies or reacts to the death of its child.

You can see if the process is a zombie by using top or the following command:

ps aux | awk '$8=="Z" {print $2}'
  • 13
    Umm, I always dislike this kind of "hard" field names with ps. Who can be sure that the required field will always be the 8th, with all implementations of ps in all Unices? – syntaxerror Feb 7 '15 at 17:15
25

Check your /var/log/kern.log and /var/log/dmesg (or equivalents) for any clues. In my experience this has happened to me only when an NFS mount's network connection has suddenly dropped or a device driver crashed. Could happen if a hard drive crashes as well, I believe.

You can use lsof to see what device files the process has open.

  • 6
    +1 for mention of NFS. A few years back this happened to me every couple of months-- if the NFS server crashed, NFS clients on all (patched) RHEL boxes would hang. kill -9 usually didn't work, even after waiting 60 minutes. The only solution was to reboot. – Stefan Lasiewski Jan 11 '11 at 17:02
17

If @Maciej's and @Gilles's answer's don't solve your problem, and you don't recognize the process (and asking what it is with your distro doesn't turn up answers ). Check for Rootkit's and any other signs that you've been owned. A rootkit is more than capable of preventing you from killing the process. In fact many are capable of preventing you from seeing them. But if they forget to modify 1 small program they might be spotted ( e.g. they modified top, but not htop ). Most likely this is not the case but better safe than sorry.

  • I guess many rootkits inserts themselves into kernel to make things simpler (no need guessing what user have and downloading MBs of patched programs). However it is still worth checking (++vote). – Maciej Piechotka Jan 12 '11 at 22:12
11

Kill actually means send a signal. there are multiple signals you can send. kill -9 is a special signal.

When sending a signal the application deals with it. if not the kernel deals with it. so you can trap a signal in your application.

But I said kill -9 was special. It is special in that the application doesn't get it. it goes straight to the kernel which then truly kills the application at the first possible opportunity. in other words kills it dead

kill -15 sends the signal SIGTERM which stands for SIGNAL TERMINATE in other words tells the application to quit. This is the friendly way to tell an application it is time to shutdown. but if the application is not responding kill -9 will kill it.

if kill -9 doesn't work it probably means your kernel is out of whack. a reboot is in order. I can't recall that ever happening.

  • 5
    15 is SIGTERM (friendly kill), not SIGHUP. SIGHUP is for the controlling terminal being closed or the communication channel being lost – JoelFan Jan 11 '11 at 5:53
11

First, check if its a Zombie process (which is very possible):

ps -Al

You will see something like:

0 Z  1000 24589     1  0  80   0 -     0 exit   ?        00:00:00 soffice.bin <defunct>

(Note the "Z" on the left)

If the 5th column is not 1, then it means it has a parent process. Try killing that parent process id.

If its PPID = 1, DON'T KILL IT!!, think which other devices or processes may be related to it.

For example, if you were using a mounted device or samba, try to unmount it. That may release the Zombie process.

NOTE: If ps -Al (or top) shows a "D" instead of "Z", it could be related to remote mount (like NFS). In my experience, rebooting is the only way to go there, but you may check the other answers which cover that case in more detail.

  • 1
    Sending SIGCHLD to the parent process may cause the parent to recognize the process has died. This should work even when the PPID = 1. This is normally sent by the kernel, but can be sent with to the parent via kill as well (kill -17 on Linux, check the manpages on other *nix). This usage of kill will not actually "kill" the parent, but rather (re)informs it that a child has died and needs to be cleaned up. Note that sigchld has to be sent to the parent of the zombie, not the zombie itself. – Stephanie Jan 21 '14 at 11:21
9

As others have mentioned, a process in uninterruptible sleep cannot be killed immediately (or, in some cases, at all). It's worth noting that another process state, TASK_KILLABLE, was added to solve this problem in certain scenarios, particularly the common case where the process is waiting on NFS. See http://lwn.net/Articles/288056/

Unfortunately I don't believe this is used anywhere in the kernel but NFS.

9

The init process is immune to SIGKILL.

This is also true also for kernel threads, i.e. "processes" with a PPID equal to 0.

  • 1
    Kernel tasks can also be immune to SIGKILL. This happens often enough with Btrfs. – Tobu Feb 28 '13 at 10:37
6

Made a little script that helped me a lot take a look!

You can use it to kill any process with a given name in its path(pay attention to this!!) Or you can kill any process of a given user using the "-u username" parameter.

#!/bin/bash

if [ "$1" == "-u" ] ; then\n
        PID=`grep "$2" /etc/passwd | cut -d ":" -f3`
        processes=`ps aux | grep "$PID" | egrep -v "PID|ps \-au|killbyname|grep" | awk '{ print $2}'`
        echo "############# Killing all processes of user: $2 ############################"
else
        echo "############# Killing processes by name: $1 ############################"
        processes=`ps aux | grep "$1" | egrep -v "killbyname|grep" | awk '{ print $2}' `
fi


for process in $processes ; do
        # "command" stores the entire commandline of the process that will be killed
        #it may be useful to show it but in some cases it is counter-productive
        #command=`ps aux | grep $process | egrep -v "grep" | awk '{ print $2 }'`
        echo "Killing process: $process"
        echo ""
        kill -9 $process
done
  • 4
    Instead of just linking to it, can you instead post the code here. – Tshepang Mar 27 '13 at 20:53
  • 3
    Add a bit of description with (or at least instead) of the code... – vonbrand Mar 27 '13 at 21:23
  • Yup but the "$name" is more aggregating... it will kill any process with "$name" in its running path. Can be very useful whan you have these huge command lines and you don't know what the process name is. – user36035 Apr 1 '13 at 17:43
5

There are cases where even if you send a kill -9 to a process, that pid will stop, but the process restarts automatically (for instance, if you try it with gnome-panel, it will restart): could that be the case here?

  • 8
    When something like this happens, the PID actually changes. So I would have noticed. – Tshepang Jan 11 '11 at 23:19
2

from here originally:

check if strace shows anything

strace -p <PID>

try attaching to the process with gdb

gdb <path to binary> <PID>

if the process was interacting with a device that you can unmount, remove the kernel module for, or physically disconnect/unplug... then try that.

  • Worked for me! (unplugging the USB device, which was hanging sublime-text) – nmz787 Mar 19 '17 at 18:18
1

I had kind of this issue. This was a program that I had launched with strace and interrupted with Ctrl+C. It ended up in a T (traced or stopped) state. I don't know how it happened exactly, but it was not killable with SIGKILL.

Long story short, I succeeded in killing it with gdb:

gdb -p <PID>
> kill
Kill the program being debugged? (y or n) y
> quit
0

Based on a clue from gilles' answer, I had a process marked "Z" ("" in ps) that was using system resources, it even had a port open that was LISTEN'ing and you could connect to it. This was after executing a kill -9 on it. Its parent was "1" (i.e. init) so theoretically it should just disappear. But it wasn't, it was sticking around, though not running.

So in my case it was zombie but still consuming resources...FWIW.

And it was not killable by kill -9.

And its parent was init but it wasn't being reaped (cleaned up). I.e. init had a zombie child.

And reboot was not necessary to fix the problem. Though a reboot "would have worked" around the problem/made it faster shutdown. Just not graceful, which was still possible.

And it was a LISTEN port owned by a zombie process (and a few other ports too like CLOSE_WAIT status connected localhost to localhost). And it still even accepted connections. Even as a zombie. I guess it hadn't gotten around to cleanup up the ports yet so incoming connections were still added to the tcp listening port's backlog, though they had no chance of being accepted.

Turns out that I had an internal thread within it that was executing a "system call" (ioctl in this instance) that was taking a few hours to return (this was expected). Apparently the system cannot kill it "all the way" until it returns from that. After a few hours it cleared up and the sockets were all automatically closed, etc. as expected. That's some languishing death time!

Also check dmesg to see if there was a kernel panic (i.e. kernel bug).

protected by Community Jun 21 '17 at 10:48

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.