systemd: restart service if any of the forked processes are killed

Question

Is there any way of auto-restart a systemd service if any of its forked processes are killed?

After creating a systemd unit file for Control-M

[Unit]

After       = network-online.target
After       = remote-fs.target
Description = Control-M agent
Wants       = network-online.target

[Service]

ExecStart   = /opt/ctmagent/ctm/scripts/start-ag -u ctmagent -p ALL
ExecStop    = /opt/ctmagent/ctm/scripts/shut-ag -u ctmagent -p ALL
Restart     = always
RestartSec  = 5
TimeoutSec  = 5min
Type        = forking

[Install]

WantedBy = multi-user.target

I noticed that it forks 3 different processes

# systemctl status ctmag.service

CGroup: /system.slice/ctmag.service
   ├─129041 /opt/ctmagent/ctm/exe/p_ctmag
   ├─129089 /opt/ctmagent/ctm/exe/p_ctmat
   └─129091 /opt/ctmagent/ctm/exe/p_ctmatw -ATW_NAME ATW000

If i kill p_ctmag, systemd restarts everything, however, if i kill p_ctmat it desn't

Or, since p_ctmat opens 127.0.0.1:7035 , can systemd monitor this port and restart the service if closed?

---

PS1: I do not have control over start-ag and shut-ag, so Systemd restart service if one of the processes in the control group fails won't help here.

PS2: Also, I can't add another script 'layer', wrapping the script in another one. Need pure systemd unit file solution.

Answer 1

The trouble you're running into is that BMC has not yet built a native systemd service file for Control-M/Agent. The installation guide for version 9.0.18.200) (login required), for systemd-systems directs you to use a service file with this template:

[Unit]
Description=Control-M Agent

[Service]
Type=forking
RemainAfterExit=yes
ExecStart=[agent_home_dir]/ctm/scripts/rc.agent_user start
ExecStop=[agent_home_dir]/ctm/scripts/rc.agent_user stop

[Install]
WantedBy=multi-user.target

... where the Exec lines are just wrappers around start-ag and shut-ag, which is what you're using. The trouble boils down to systemd being told that this is a forking Type of service, when in fact the "service" (start script) starts multiple processes. systemd considers the (forking) service to be successful when it sees the first child process after the initial script. That process ends up being p_ctmag, as you've seen. The systemctl status output makes it clear:

Process: 6519 ExecStart=/opt/ctmagent/ctm/scripts/start-ag -u ctmagent -p ALL (code=exited, status=0/SUCCESS)

... where pid 6519 was the start-ag script, which has since exited. This is followed by the remaining control-group processes you see.

I'd additionally point out here that the systemd docs say:

If this setting is used, it is recommended to also use the PIDFile= option, so that systemd can reliably identify the main process of the service.

... which the suggested BMC service file does not include, despite there being PID files for all three processes under $CONTROLM/pid/; though they're not real PID files, since the files are named for the pid instead of the pid being their contents.

This is why systemd is not restarting the p_ctmat process.

Short of rewriting a thousand-plus line vendor script to split the processes into individually-controlled ones (complete with the two handfuls of environment variables those scripts set) or waiting for BMC to provide a useful systemd script, my suggestion would be to change your agent configuration to set the WATCHDOG_ENABLED parameter to Y. In my testing, this enables the p_ctmag process to watch the p_ctmat process, which will restart it if necessary. The p_ctmatw process is a child of the p_ctmat process, which will restart p_ctmatw as needed.

From there, you're covered:

• p_ctmag will be restarted by systemd, as now
• p_ctmat will be restarted by p_ctmag (with WATCHDOG_ENABLED Y)
• p_ctmatw will be restarted by p_ctmat, as now