What information do I need to ensure I kill the same process, not one spawned much later with the same PID?

Question

I have a bunch of commands that I start with a start.sh script, storing their PID. Then later, I want to stop them with a stop.sh run at the user's convenience.

Watch the trap:

1. I run start.sh today, which stores in a file the PIDs 15000, 15001, 15002.
2. I forgot stopping my processes. And a week after, I rebooted my computer.
3. I run the stop.sh script, now. It attempts to kill the tasks with PIDs 15000, 15001, 15002, reading them without thinking from the file.
   => These tasks, if some happen to have these PIDs on my new rebooted system, are no longer the ones I started by my start.sh script, and I will put my system into an unknown state.

How, when I catch first the PID of a process with a $$ in a Linux script, may I gather other information to ensure I can have no confusion with another task of same PID that could appear in the future?

Gathering PPID, for example, or start date/time, or something that ensures some kind of "universal uniqueness", if I can write this.. ?

How do you gather process information and how do you kill it without confusion?

Answer 1

How do you gather process information and how do you kill it without confusion?

You don’t.

Instead, you ensure the process runs in a context that you can reliably reference later to find it again.

The correct way to handle this is to use your target platform’s service management system (usually systemd on Linux systems these days). They will handle things correctly in a majority of cases and are specifically designed to do this type of thing.

Alternatives, in decreasing order of preference, are:

• Use a cgroup with a specific name. This approach is Linux-specific, but has a number of distinct benefits, such as being able to reliably and atomically kill all the children of the processes you start. They inherently solve the lifetime issue because the specific processes are explicitly associated with the cgroup, instead of the PIDs being associated with the cgroup.
• Use a supervisor system such as runit, s6, or daemontools. These solve the issue by utilizing a process that can be easily and reliably located to act as a parent for the process you want to monitor.
• Put your PID files in /run, where they should be. The issue you point out of PID reuse across system reboots is a known issue that’s been reliably solved for decades by simply putting PID files in a directory that gets wiped each time the system reboots. /run is the standard location for this on Linux systems. This still has issues with PID reuse (because PIDs are only unique for the lifetime of the associated processes, so one of your processes dying unexpectedly and leaving a PID file behind may still run into a reuse problem).

Answer 2

When I had to launch all kind of diagnostic scripts across a large network, I had all my scripts accept (and ignore) a --tag=.... option. (Obviously, you cannot do this with standard commands, but you can wrap them in a parent shell).

A typical --tag would contain (at least) the hostname from which it was launched, a random number, and the time of launch (down to nanosecond accuracy). For a remote task, you may not even know the pid on the remote system.

The ps command can show the args so you can grep a specific process. You could even have a cron job that produced a report of latent processes at intervals, and weeded out from your list those that had terminated.

Answer 3

Store the PID information on a tmpfs filesystem, so after reboot, the files do not exist.

/run is usually tmpfs, or /tmp on some distros is tmpfs.

or mount your own

# mount tmpfs /path/to/your/mountpoint -t tmpfs 

Answer 4

You could gather additional information on the processes, but strictly speaking, it's hard to be absolutely sure the process you're going to kill is the same as the one you think it is.

Even disregarding reboots, it's possible that just before you kill the process, it dies of an unrelated reason, and another process starts at just the right time to get the same PID again. Any checks would still be racy, as it's possible for the process to be replaced with another between the check and the sending of the terminating signal.

However, if you take care to start the process under an UID that is used for nothing else, and send the terminating signal from a process with that same UID (and not root), you'll know you won't succeed in killing anything else. (The PID could still be reused by some unrelated process, then you just couldn't kill the new one.)

If do go the way of checking if the process is the same (and can ignore the chance of a race), you should probably at least check the process name / full command line of the process you're going to kill. E.g. run something like pgrep somecmd, or pgrep -f 'somecmd with some args' to find processes with the given name / command line, and see if your PID is listed in the output.

Answer 5

Have start.sh bind-mount /proc/15000 to some other directory somewhere. (Preferably this would be done by the parent process before it exits or waits to avoid a race condition.) In stop.sh, try to open the directory you bind-mounted. If the bind-mount is gone, then the system got rebooted. If opening the directory fails with ESRCH (No such process), then the process exited. (This will happen even if a new process with the same PID has since started running.) If opening the directory succeeds and the bind-mount is still there, then it's still your process and it's safe to kill. (To avoid yet another race condition, prefer pidfd_send_signal to kill it.)

Answer 6

You also need to deal with PID rollover - I believe the default on Linux is 32768 - hence if you execute enough processes the next one will rollover.

Note: The rollover bypasses any processes that already exist such as (PID 1) init.

However if you store both the PID and the start time of the process that should be sufficient to identify the process uniquely on this machine.

If you need uniqueness across machines you will also need to store something that uniquely identifies the machine.