3

Is there any way of auto-restart a systemd service if any of its forked processes are killed?

After creating a systemd unit file for Control-M

[Unit]

After       = network-online.target
After       = remote-fs.target
Description = Control-M agent
Wants       = network-online.target

[Service]

ExecStart   = /opt/ctmagent/ctm/scripts/start-ag -u ctmagent -p ALL
ExecStop    = /opt/ctmagent/ctm/scripts/shut-ag -u ctmagent -p ALL
Restart     = always
RestartSec  = 5
TimeoutSec  = 5min
Type        = forking

[Install]

WantedBy = multi-user.target

I noticed that it forks 3 different processes

# systemctl status ctmag.service

CGroup: /system.slice/ctmag.service
   ├─129041 /opt/ctmagent/ctm/exe/p_ctmag
   ├─129089 /opt/ctmagent/ctm/exe/p_ctmat
   └─129091 /opt/ctmagent/ctm/exe/p_ctmatw -ATW_NAME ATW000

If i kill p_ctmag, systemd restarts everything, however, if i kill p_ctmat it desn't

Or, since p_ctmat opens 127.0.0.1:7035 , can systemd monitor this port and restart the service if closed?


PS1: I do not have control over start-ag and shut-ag, so Systemd restart service if one of the processes in the control group fails won't help here.

PS2: Also, I can't add another script 'layer', wrapping the script in another one. Need pure systemd unit file solution.

9
  • Could you explain a bit why you cannot use a helper/wrapper script? That probably rules out most of the obvious solutions.
    – TooTea
    Feb 14, 2019 at 13:01
  • 2
    Note that start-ag and shut-ag are vendor-provided wrapper scripts already. The start-ag script (among a million other things) eventually calls nohup some/binary & for the various processes.
    – Jeff Schaller
    Feb 14, 2019 at 13:52
  • When I kill p_ctmat here, the AG process restarts it (.../dailylog/daily_ctmag_*.log contains "WATCHDOG OF PROCESS AG STARTED THE PROCESS AT"). Does that not happen for you?
    – Jeff Schaller
    Feb 14, 2019 at 14:07
  • 1
    The version 9 doco says rc.agent_user start and rc.agent_user stop. It's still an unnecessary wrapper script, though. (-:
    – JdeBP
    Feb 14, 2019 at 14:14
  • 3
    Side note: BMC is stil getting used to the idea of systemd.
    – Jeff Schaller
    Feb 14, 2019 at 14:16

1 Answer 1

3

The trouble you're running into is that BMC has not yet built a native systemd service file for Control-M/Agent. The installation guide for version 9.0.18.200) (login required), for systemd-systems directs you to use a service file with this template:

[Unit]
Description=Control-M Agent

[Service]
Type=forking
RemainAfterExit=yes
ExecStart=[agent_home_dir]/ctm/scripts/rc.agent_user start
ExecStop=[agent_home_dir]/ctm/scripts/rc.agent_user stop

[Install]
WantedBy=multi-user.target

... where the Exec lines are just wrappers around start-ag and shut-ag, which is what you're using. The trouble boils down to systemd being told that this is a forking Type of service, when in fact the "service" (start script) starts multiple processes. systemd considers the (forking) service to be successful when it sees the first child process after the initial script. That process ends up being p_ctmag, as you've seen. The systemctl status output makes it clear:

Process: 6519 ExecStart=/opt/ctmagent/ctm/scripts/start-ag -u ctmagent -p ALL (code=exited, status=0/SUCCESS)

... where pid 6519 was the start-ag script, which has since exited. This is followed by the remaining control-group processes you see.

I'd additionally point out here that the systemd docs say:

If this setting is used, it is recommended to also use the PIDFile= option, so that systemd can reliably identify the main process of the service.

... which the suggested BMC service file does not include, despite there being PID files for all three processes under $CONTROLM/pid/; though they're not real PID files, since the files are named for the pid instead of the pid being their contents.

This is why systemd is not restarting the p_ctmat process.

Short of rewriting a thousand-plus line vendor script to split the processes into individually-controlled ones (complete with the two handfuls of environment variables those scripts set) or waiting for BMC to provide a useful systemd script, my suggestion would be to change your agent configuration to set the WATCHDOG_ENABLED parameter to Y. In my testing, this enables the p_ctmag process to watch the p_ctmat process, which will restart it if necessary. The p_ctmatw process is a child of the p_ctmat process, which will restart p_ctmatw as needed.

From there, you're covered:

  • p_ctmag will be restarted by systemd, as now
  • p_ctmat will be restarted by p_ctmag (with WATCHDOG_ENABLED Y)
  • p_ctmatw will be restarted by p_ctmat, as now
2
  • Be sure to skip the first and third instructions in that document for "Control-M/Agent automatic startup procedures", as they both point you towards the SysV init.d structure.
    – Jeff Schaller
    Feb 22, 2019 at 15:59
  • you nailed it. WATCHDOG_ENABLED was the culprit.
    – RASG
    Feb 22, 2019 at 17:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .