Grouping services so that failure in one restarts them all

Question

I have a group of services that are fundamentally linked (ideally they should be ran as a single service/process but that work still needs to be done) in that I want the failure of one service to cause the other services to be restarted as well.

I'm using a target unit to start them all at the same time and I have them all set to Restart=always to individually recover but that wouldn't restart the other services if one service failed for any reason.

I thought I might get away with using the BindsTo directive which does seem to stop all the services if a single one fails (such as the process being killed).

My service files currently look like this:

[Unit]
Description=foo
After=syslog.target network.target
BindsTo=foo.service bar.service baz.service all-services.target

[Service]
User=foo
WorkingDirectory=~
Restart=always
RestartSec=90
StartLimitInterval=400
StartLimitBurst=3
SyslogIdentifier=foo
ExecStart=/path/to/binary 
[Install]
WantedBy=all-services.target

With a grouping target defined as:

[Unit]
Description=all-services
BindsTo=foo.service bar.service baz.service

But the BindsTo directive seems to completely stop all the services when one fails and never attempt a restart.

I can't seem to find anything that suggests this service group restarting property is achievable directly in systemd rather than restarting a chain of dependencies but I might be missing something (or ignoring a better way of doing this).

Answer 1

Using the PartOf directive along with the target unit to start all of the services as a group seems to achieve my goals.

My service files now look like this:

[Unit]
Description=foo
After=syslog.target network.target
PartOf=foo.service bar.service baz.service all-services.target

[Service]
User=foo
WorkingDirectory=~
Restart=always
RestartSec=90
StartLimitInterval=400
StartLimitBurst=3
SyslogIdentifier=foo
ExecStart=/path/to/binary 
[Install]
WantedBy=all-services.target

and the grouping target looks like:

[Unit]
Description=all-services
PartOf=foo.service bar.service baz.service

I can start and restart foo, bar and baz services by running systemctl start all-services.target and killing the process used by foo causes them all to be stopped and restarted.

One thing that isn't clear is if I run systemctl stop all-services.target it will stop all the services but a systemctl start all-services.target from the stopped state only starts the target unit instead of all of the services.

I've tried swapping out WantedBy to RequiredBy in the service units but that doesn't seem to change anything.

Answer 2

For your requirement "failure in one restarts them all", a bidirectional PartOf between the services and the target is sufficient.

Services:

[Unit]
PartOf=all-services.target

[Service]
ExecStart=/path/to/binary
Restart=on-failure

Target:

[Unit]
PartOf=foo.service bar.service baz.service

In case a unit fails and restarts, the target will broadcast the restart to all services.

An additional PartOf between the services is not necessary. It would only be relevant in cases, when the target is not active.

This answer gives an explanation, why BindsTo is not working in your case.

Your foo unit is using BindsTo=, which is meant to track the state of a unit that may disappear.

So the foo unit will become inactive before it restarts.