39

Suppose, for example, you have a shell script similar to:

longrunningthing &
p=$!
echo Killing longrunningthing on PID $p in 24 hours
sleep 86400
echo Time up!
kill $p

Should do the trick, shouldn't it? Except that the process may have terminated early and its PID may have been recycled, meaning some innocent job get a bomb in its signal queue instead. In practice this possibly does matter, but its worrying me nonetheless. Hacking longrunningthing to drop dead by itself, or keep/remove its PID on the FS would do but I'm thinking of the generic situation here.

  • 3
    You make sure that if your targeted process dies, it kills its killer in the process. – mikeserv Jan 25 '15 at 18:24
  • 2
    Use killall which matches on name, so at least you are only killing a process with the same name as longrunningthing. Assuming you will only have one of these running at a time. – LawrenceC Jan 25 '15 at 23:29
  • 8
    You can save the start time of the original process and, before killing, check that the start time of the process with that pid matches what you saved. The pair pid, start-time is a unique identifier for the processes in Linux. – Bakuriu Jan 26 '15 at 10:43
  • 1
    May I ask why you need this? what is the fundamental thing you are trying to achieve? (something that run continuously but is reseted every 24h?) – Olivier Dulac Jan 26 '15 at 17:09
  • @mikeserv A process can't guarantee that something will happen in the event of its own death. – kasperd Jan 26 '15 at 22:34
29

Best would be to use the timeout command if you have it which is meant for that:

timeout 86400 cmd

The current (8.23) GNU implementation at least works by using alarm() or equivalent while waiting for the child process. It does not seem to be guarding against the SIGALRM being delivered in between waitpid() returning and timeout exiting (effectively cancelling that alarm). During that small window, timeout may even write messages on stderr (for instance if the child dumped a core) which would further enlarge that race window (indefinitely if stderr is a full pipe for instance).

I personally can live with that limitation (which probably will be fixed in a future version). timeout will also take extra care to report the correct exit status, handle other corner cases (like SIGALRM blocked/ignored on startup, handle other signals...) better than you'd probably manage to do by hand.

As an approximation, you could write it in perl like:

perl -MPOSIX -e '
  $p = fork();
  die "fork: $!\n" unless defined($p);
  if ($p) {
    $SIG{ALRM} = sub {
      kill "TERM", $p;
      exit 124;
    };
    alarm(86400);
    wait;
    exit (WIFSIGNALED($?) ? WTERMSIG($?)+128 : WEXITSTATUS($?))
  } else {exec @ARGV}' cmd

There's a timelimit command at http://devel.ringlet.net/sysutils/timelimit/ (predates GNU timeout by a few months).

 timelimit -t 86400 cmd

That one uses an alarm()-like mechanism but installs a handler on SIGCHLD (ignoring stopped children) to detect the child dying. It also cancels the alarm before running waitpid() (that doesn't cancel the delivery of SIGALRM if it was pending, but the way it's written, I can't see it being a problem) and kills before calling waitpid() (so can't kill a reused pid).

netpipes also has a timelimit command. That one predates all the other ones by decades, takes yet another approach, but doesn't work properly for stopped commands and returns a 1 exit status upon timeout.

As a more direct answer to your question, you could do something like:

if [ "$(ps -o ppid= -p "$p")" -eq "$$" ]; then
  kill "$p"
fi

That is, check that the process is still a child of ours. Again, there's a small race window (in between ps retrieving the status of that process and kill killing it) during which the process could die and its pid be reused by another process.

With some shells (zsh, bash, mksh), you can pass job specs instead of pids.

cmd &
sleep 86400
kill %
wait "$!" # to retrieve the exit status

That only works if you spawn only one background job (otherwise getting the right jobspec is not always possible reliably).

If that's an issue, just start a new shell instance:

bash -c '"$@" & sleep 86400; kill %; wait "$!"' sh cmd

That works because the shell removes the job from the job table upon the child dying. Here, there should not be any race window since by the time the shell calls kill(), either the SIGCHLD signal has not been handled and the pid can't be reused (since it has not been waited for), or it has been handled and the job has been removed from the process table (and kill would report an error). bash's kill at least blocks SIGCHLD before it accesses its job table to expand the % and unblocks it after the kill().

Another option to avoid having that sleep process hanging around even after cmd has died, with bash or ksh93 is to use a pipe with read -t instead of sleep:

{
  {
    cmd 4>&1 >&3 3>&- &
    printf '%d\n.' "$!"
  } | {
    read p
    read -t 86400 || kill "$p"
  }
} 3>&1

That one still has race conditions, and you lose the command's exit status. It also assumes cmd doesn't close its fd 4.

You could try implementing a race-free solution in perl like:

perl -MPOSIX -e '
   $p = fork();
   die "fork: $!\n" unless defined($p);
   if ($p) {
     $SIG{CHLD} = sub {
       $ss = POSIX::SigSet->new(SIGALRM); $oss = POSIX::SigSet->new;
       sigprocmask(SIG_BLOCK, $ss, $oss);
       waitpid($p,WNOHANG);
       exit (WIFSIGNALED($?) ? WTERMSIG($?)+128 : WEXITSTATUS($?))
           unless $? == -1;
       sigprocmask(SIG_UNBLOCK, $oss);
     };
     $SIG{ALRM} = sub {
       kill "TERM", $p;
       exit 124;
     };
     alarm(86400);
     pause while 1;
   } else {exec @ARGV}' cmd args...

(though it would need to be improved to handle other types of corner cases).

Another race-free method could be using process groups:

set -m
((sleep 86400; kill 0) & exec cmd)

However note that using process groups can have side-effects if there's I/O to a terminal device involved. It has the additional benefit though to kill all the other extra processes spawned by cmd.

  • 4
    Why not mention the best method first? – deltab Jan 26 '15 at 3:43
  • 2
    @deltab: timeout is not portable, the answer mentioned a portable solution first. – cuonglm Jan 26 '15 at 4:51
  • 1
    @deltab: it gives insight on how things work and especially how "common sense" approach can fail (Stephane prefers to teach one to fish first, which I like). One is expected to read the whole answer – Olivier Dulac Jan 26 '15 at 13:10
  • @Stephane: for the "getting the right jobspec is not always possible reliably" : can't you first count the output of jobs and then know that (as it's your own shell, in which you have control on what happens next) the next backgrounded job will be N+1 ? [then you can save N, and later kill %N+1]) – Olivier Dulac Jan 26 '15 at 13:13
  • 1
    @OlivierDulac, that would assume no past job has terminated by the time you start a new one (shells reuse job numbers). – Stéphane Chazelas Jan 26 '15 at 13:44
27

In general, you can't. All of the answers given so far are buggy heuristics. There is only one case in which you can safely use the pid to send signals: when the target process is a direct child of the process that will be sending the signal, and the parent has not yet waited on it. In this case, even if it has exited, the pid is reserved (this is what a "zombie process" is) until the parent waits on it. I'm not aware of any way to do that cleanly with the shell.

An alternative safe way to kill processes is to start them with a controlling tty set to a pseudo-terminal for which you own the master side. You can then send signals via the terminal, e.g. writing the character for SIGTERM or SIGQUIT over the pty.

Yet another way that's more convenient with scripting is to use a named screen session and send commands to the screen session to end it. This process takes place over a pipe or unix socket named according to the screen session, which won't automatically be reused if you choose a safe unique name.

  • 4
    I can't see why that couldn't be done in shells. I've given several solutions. – Stéphane Chazelas Jan 26 '15 at 12:40
  • 3
    Could you please give an explanation and some sort of quantitative discussion of race windows and other drawbacks? Without that, "All of the answers given so far are buggy heuristics" is a just unnecessarily confrontational without any benefits. – peterph Jan 26 '15 at 13:18
  • 2
    @peterph: In general, any use of a pid is a TOCTOU race -- no matter how you check whether it still refers to the same process you expect it to refer to, it could cease to refer to that process and refer to some new process in the interval before you use it (sending the signal). The only way to prevent this is to be able to block the freeing/reuse of the pid, and the only process that can do this is the direct parent. – R.. Jan 26 '15 at 18:17
  • 1
    @StéphaneChazelas: How do you prevent the shell from waiting on a pid of a background process that has exited? If you can do that, the problem is easily solvable in the case OP needs. – R.. Jan 26 '15 at 18:18
  • 4
    @peterph: "The race window is small" is not a solution. And the rarity of the race is dependent on sequential pid assignment. Bugs that cause something very bad to happen once in a year are much worse than bugs that happen all the time because they're virtually impossible to diagnose and fix. – R.. Jan 26 '15 at 19:10
10
  1. When launching the process save its start time:

    longrunningthing &
    p=$!
    stime=$(TZ=UTC0 ps -p "$p" -o lstart=)
    
    echo "Killing longrunningthing on PID $p in 24 hours"
    sleep 86400
    echo Time up!
    
  2. Before trying to kill the process stop it (this isn't truly essential, but it's a way to avoid race conditions: if you stop the process, it's pid cannot be reused)

    kill -s STOP "$p"
    
  3. Check that the process with that PID has the same start time and if yes, kill it, otherwise let the process continue:

    cur=$(TZ=UTC0 ps -p "$p" -o lstart=)
    
    if [ "$cur" = "$stime" ]
    then
        # Okay, we can kill that process
        kill "$p"
    else
        # PID was reused. Better unblock the process!
        echo "long running task already completed!"
        kill -s CONT "$p"
    fi
    

This works because there can be only one process with the same PID and start time on a given OS.

Stopping the process during the check makes race-conditions a non-issue. Obviously this has the problem that, some random process may be stopped for some milliseconds. Depending on the type of process this may or may not be an issue.


Personally I'd simply use python and psutil which handles PID reuse automatically:

import time

import psutil

# note: it would be better if you were able to avoid using
#       shell=True here.
proc = psutil.Process('longrunningtask', shell=True)
time.sleep(86400)

# PID reuse handled by the library, no need to worry.
proc.terminate()   # or: proc.kill()
  • Python rules in UNIX... I'm not sure why more answers don't start there as I'm sure most systems do not prohibit the use of it. – Mr. Mascaro Jan 26 '15 at 18:03
  • I've used a similar scheme (using the start time) before, but your sh scripting skills are neater than mine! Thanks. – FJL Jan 26 '15 at 20:14
  • That means you're potentially stopping the wrong process. Note that ps -o start= format changes from 18:12 to Jan26 after a while. Beware of DST changes as well. If on Linux, you'll probably prefer TZ=UTC0 ps -o lstart=. – Stéphane Chazelas Jan 26 '15 at 20:30
  • @StéphaneChazelas Yes, but you let it continue afterwards. I explicitly said: depending on the type of task that that process is performing you may have some trouble for stopping it some milliseconds. Thanks for the tip on lstart, I'll edit it in. – Bakuriu Jan 26 '15 at 20:33
  • Note that (unless your system limits the number of processes per user), it's easy for anyone to fill the process table with zombies. Once there are only 3 available pids left, it's easy for anyone to start hundreds of different processes with the same pid within a single second. So, strictly speaking, your "there can be only one process with the same PID and start time on a given OS" is not necessarily true. – Stéphane Chazelas Jan 26 '15 at 21:19
7

On a linux system you can assure that a pid will not be reused by keeping its pid namespace alive. This can be done via the /proc/$pid/ns/pid file.

  • man namespaces -

    Bind mounting (see mount(2)) one of the files in this directory to somewhere else in the filesystem keeps the corresponding namespace of the process specified by pid alive even if all processes currently in the namespace terminate.

    Opening one of the files in this directory (or a file that is bind mounted to one of these files) returns a file handle for the corresponding namespace of the process specified by pid. As long as this file descriptor remains open, the namespace will remain alive, even if all processes in the namespace terminate. The file descriptor can be passed to setns(2).

You can isolate a group of processes - basically any number of processes - by namespacing their init.

  • man pid_namespaces -

    The first process created in a new namespace (i.e., the process created using clone(2) with the CLONE_NEWPID flag, or the first child created by a process after a call to unshare(2) using the CLONE_NEWPID flag) has the PID 1, and is the init process for the namespace (see init(1)). A child process that is orphaned within the namespace will be reparented to this process rather than init(1) (unless one of the ancestors of the child in the same PID namespace employed the prctl(2) PR_SET_CHILD_SUBREAPER command to mark itself as the reaper of orphaned descendant processes).

    If the init process of a PID namespace terminates, the kernel terminates all of the processes in the namespace via a SIGKILL signal. This behavior reflects the fact that the init process is essential for the correct operation of a PID namespace.

The util-linux package provides many useful tools for manipulating namespaces. For example, there is unshare, though, if you have not already arranged for its rights in a user namespace, it will require superuser rights:

unshare -fp sh -c 'n=
    echo "PID = $$"
    until   [ "$((n+=1))" -gt 5 ]
    do      while   sleep 1
            do      date
            done    >>log 2>/dev/null   &
    done;   sleep 5' >log
cat log; sleep 2
echo 2 secs later...
tail -n1 log

If you have not arranged for a user namespace, then you can still safely execute arbitrary commands by immediately dropping privileges. The runuser command is another (non setuid) binary provided by the util-linux package and incorporating it might look like:

sudo unshare -fp runuser -u "$USER" -- sh -c '...'

...and so on.

In the above example two switches are passed to unshare(1) the --fork flag which makes the invoked sh -c process the first child created and ensures its init status, and the --pid flag which instructs unshare(1) to create a pid namespace.

The sh -c process spawns five backgrounded child shells - each an inifinite while loop that will continue to append the output of date to the end of log for as long as sleep 1 returns true. After spawning these processes sh calls sleep for an additional 5 seconds then terminates.

It maybe worth noting that if the -f flag were not used none of the backgrounded while loops would terminate, but with it...

OUTPUT:

PID = 1
Mon Jan 26 19:17:45 PST 2015
Mon Jan 26 19:17:45 PST 2015
Mon Jan 26 19:17:45 PST 2015
Mon Jan 26 19:17:45 PST 2015
Mon Jan 26 19:17:45 PST 2015
Mon Jan 26 19:17:46 PST 2015
Mon Jan 26 19:17:46 PST 2015
Mon Jan 26 19:17:46 PST 2015
Mon Jan 26 19:17:46 PST 2015
Mon Jan 26 19:17:46 PST 2015
Mon Jan 26 19:17:47 PST 2015
Mon Jan 26 19:17:47 PST 2015
Mon Jan 26 19:17:47 PST 2015
Mon Jan 26 19:17:47 PST 2015
Mon Jan 26 19:17:47 PST 2015
Mon Jan 26 19:17:48 PST 2015
Mon Jan 26 19:17:48 PST 2015
Mon Jan 26 19:17:48 PST 2015
Mon Jan 26 19:17:48 PST 2015
Mon Jan 26 19:17:48 PST 2015
2 secs later...
Mon Jan 26 19:17:48 PST 2015
  • Interesting answer that seem to be robust. Probably a bit overkill for basic usage, but worth having a thought. – Uriel Sep 12 '15 at 8:42
  • I do not see how or why keeping a PID namespace alive prevents reuse of a PID. The very manpage you quote - As long as this file descriptor remains open, the namespace will remain alive, even if all processes in the namespace terminate - suggests that the processes may still terminate (and thus presumably have their process ID recycled). What does keeping the PID namespace alive have to do with preventing the PID itself from being re-used by another process? – davmac Jun 24 '16 at 14:05
5

Consider making your longrunningthing behave a bit better, a little bit more daemon-like. For example you may make it create a pidfile that will allow at least some limited control of the process. There are several ways of doing this without modifying the original binary, all involving a wrapper. For example:

  1. a simple wrapper script that will start the required job in background (with optional output redirection), write the PID of this process into a file, then wait for the process to finish (using wait) and remove the file. If during the wait the process gets killed e.g. by something like

    kill $(cat pidfile)
    

    the wrapper will just make sure the pidfile is removed.

  2. a monitor wrapper, that will put its own PID somewhere and catch (and respond to) signals sent to it. Simple example:

    #!/bin/bash
    p=0
    trap killit USR1

    killit () {
        printf "USR1 caught, killing %s\n" "$p"
        kill -9 $p
    }

    printf "monitor $$ is waiting\n"
    therealstuff &
    p=%1
    wait $p
    printf "monitor exiting\n"

Now, as @R.. and @StéphaneChazelas pointed out, these approaches often have a race condition somewhere or impose a restriction on the number of processes you can spawn. In addition, it doesn't handle the cases, where the longrunningthing may fork and the children get detached (which is probably not what was the problem in the original question).

With recent (read a couple of years old) Linux kernels, this can be nicely treated by using cgroups, namely the freezer - which, I suppose, is what some modern Linux init systems use.

  • Thanks, and to all. I'm reading everyting now.The point about longrunningthing is that you have no control over what it is. I also gave a shell script example because it explained the problem. I like yours, and all the other creative solutions here, but if you're using Linux/bash there's a "timeout" builtin for that. I suppose I ought to get the source to that and see how it does it! – FJL Jan 26 '15 at 20:11
  • @FJL, timeout is not a shell builtin. There have been various implementations of a timeout command for Linux, one was recently (2008) added to GNU coreutils (so not Linux specific), and that's what most Linux distributions use nowadays. – Stéphane Chazelas Jan 26 '15 at 20:21
  • @ Stéphane - Thanks - I subsequently found reference to GNU coreutils. They may be portable, but unless its in the base system it can't be relied on. I'm more interested in knowing how it work, although I note your comment elsewhere suggesting it's not 100% reliable. Given the way this thread has gone, I'm not surprised! – FJL Jan 27 '15 at 22:39
1

If you are running on Linux (and a few other *nixes), you can check if the process you intend to kill is still used and that the command line matches your long process. Something like :

echo Time up!
grep -q longrunningthing /proc/$p/cmdline 2>/dev/null
if [ $? -eq 0 ]
then
  kill $p
fi

An alternative can be to check for how long the process you intend to kill is running, with something like ps -p $p -o etime=. You could do it yourself by extracting this information from /proc/$p/stat, but this would be tricky (time is measured in jiffies, and you will have to use the system uptime in /proc/stat too).

Anyway, you usually cannot ensure that the process is not replaced after your check and before you kill it.

  • That's still not correct because it doesn't get rid of the race condition. – strcat Sep 10 '15 at 22:12
  • @strcat Indeed, No warranty of success, but most scripts do not even bother to do such check and only bluntly kill a cat pidfile result. I cannot remember of a clean way to do it in shell only. The proposed namespace answer seems an intersting one however... – Uriel Sep 12 '15 at 8:39
-1

This is actually a very good question.

The way to determine process uniqueness is to look at (a) where it is in memory; and (b) what that memory contains. To be specific, we want to know where in memory is the program text for the initial invocation, because we know that the text area of each thread will occupy a different location in memory. If the process dies and another one is launched with the same pid, the program text for the new process will not occupy the same place in memory and will not contain the same information.

So, immediately after launching your process, do md5sum /proc/[pid]/maps and save the result. Later when you want to kill the process, do another md5sum and compare it. If it matches, then kill the pid. If not, don't.

to see this for yourself, launch two identical bash shells. Examine the /proc/[pid]/maps for them and you will find that they are different. Why? Because even though it is the same program, they occupy different locations in memory and the addresses of their stack are different. So, if your process dies and its PID is reused, even by the same command being relaunched with the same arguments, the "maps" file will be different and you will know that you are not dealing with the original process.

See: proc man page for details.

Note that the file /proc/[pid]/stat already contains all of the information that other poster's have mentioned in their answers: age of the process, parent pid, etc. This file contains both static information and dynamic information, so if you prefer to use this file as a base of comparison, then upon launching your longrunningthing, you need to extract the following static fields from the stat file and save them for comparison later:

pid, filename, pid of parent, process group id, controlling terminal, time process started after system boot, resident set size, the address of the start of the stack,

taken together, the above uniquely identify the process, and so this represents another way to go. Actually you could get away with nothing more than "pid" and "time process started after system boot" with high degree of confidence. Simply extract these fields from the stat file and save it somewhere upon launching your process. Later before killing it, extract it again and compare. If they match, then you are assured you are looking at the original process.

  • 1
    That generally won't work as /proc/[pid]/maps changes over time as extra memory is allocate or the stack grows or new files are mmapped... And what does immediately after launching mean? After all the libraries have been mmapped? How do you determine that? – Stéphane Chazelas Jan 27 '15 at 11:26
  • I'm doing a test now on my system with two processes, one a java app and the other a cfengine server. Every 15 minutes I do md5sum on their maps files. I'll let it run for a day or two and report back here with the results. – Michael Martinez Jan 28 '15 at 0:13
  • @StéphaneChazelas: I've been checking my two processes for 16 hours now, and there's been no change in the md5sum – Michael Martinez Jan 28 '15 at 18:12
-1

Another way would be to check the age of the process before killing it. That way, you can make sure you are not killing a process that is not spawned in less than 24 hours. You can add an if condition based on that before killing the process.

if [[ $(ps -p $p -o etime=) =~ 1-. ]] ; then
    kill $p
fi

This if condition will check whether the process ID $p is less than 24 hours (86400 seconds).

P.S:- The command ps -p $p -o etime= will have the format <no.of days>-HH:MM:SS

  • The mtime of /proc/$p has nothing to do with the start time of the process. – Stéphane Chazelas Jan 27 '15 at 11:58
  • Thanks @StéphaneChazelas. You are right. I have edited the answer to change the if condition. Please feel free to comment if its buggy. – Sree Jan 30 '15 at 16:18
-3

What I do is, after having killed the process, do it again. Every time I do that the answer comes back, "no such process"

allenb   12084  5473  0 08:12 pts/4    00:00:00 man man
allenb@allenb-P7812 ~ $ kill -9 12084
allenb@allenb-P7812 ~ $ kill -9 12084
bash: kill: (12084) - No such process
allenb@allenb-P7812 ~ $ 

Couldn't be simpler and I've been doing this for years without any problems.

  • That's answering the "how can I make it worse" question, not the "how can I fix it". – Stéphane Chazelas Feb 25 '16 at 16:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.