There's a difference between the "daemon" failing and the "service" being in a failed state.
The daemon failing simply means if the program exited somehow other than returning 0 as an exit code. That's what Restart=on-failure
is referring to i.e. "Restart the daemon if it fails" Also, you can restart on other conditions see the table under Restart= which also explains what "on-failure" exactly means.
According to your service file, there is no definition of "service" failure, and I think the default definition is basically "the daemon exited somehow other than returning 0 as an exit code AND it is not currently running", but your service file has the daemon restarted immediately. So it probably does enter the "failed" state, but so briefly you don't notice it. You can check the log with journalctl -u foo.service
.
Also, you should check to see if the PID of the main process is still the same as it was +30 seconds ago. If the PID is different, that means systemd is killing the daemon, because it never called sd_notify
and then systemd is restarting the daemon very quickly because your service file says to restart it on failure, and doesn't tell it to wait between restarts (see below).
If you want your service to have a definition of failure you want to use something like:
StartLimitInterval=5min
StartLimitBurst=4
Those options are documented in depth here, however what these two basically mean in this example is "Start the daemon no more than 4 times in a 5 minute interval. If it is started, and then fails, that many times in that interval, the service is considered to be in a failed state". If that happens, then the StartLimitAction=
is taken, by default that is none.
Another configurable value is RestartSec=
. From the documentation:
Configures the time to sleep before restarting a service (as configured with
Restart=). Takes a unit-less value in seconds, or a time span value such as "5min >20s". Defaults to 100ms.
Which basically means that systemd must wait at least RestartSec=
before starting the daemon again.
So you can combine all of these options, but let's say you use these values:
StartLimitInterval=5min
StartLimitBurst=4
RestartSec=2min
Then the service could never enter the failed state, since the definition of failure is:
If in a given 5 minute interval, the service has been started, and
then failed 4 times.
But systemd won't re-start the service for two minutes after the last failure, so it would never reach the StartLimitBurst
inside of the StartLimitInterval
.