Systemd: service doesn't restart with WatchdogSec option set

Question

I'm trying to manager qemu with systemd. If qemu crashes, it can be restart automatically. I have the following unit file:

[Unit]
Description=vm manager
After=network.target
Before=shutdown.target reboot.target poweroff.target halt.target

[Service]
Type=forking
ExecStart=/root/vm/vm-manager.sh start-vm
ExecStop=/root/vm/vm-manager.sh stop-vm
KillSignal=SIGCONT
PIDFile=/root/vm/run/pid
WatchdogSec=30s
Restart=on-failure

[Install]
WantedBy=multi-user.target

I didn't call sd_notify(0, "WATCHDOG=1") in my application, but the service didn't move to 'failed' state after 30 seconds. I have two questions:

1. In my opinion, this service should be restarted after 30 seconds, why it keeps running until I kill or stop it?
2. When I kill qemu manually(I take qemu process as the main process), the service restart immediately, without waiting.

Besides the two questions, if there's anything wrong or suggestion about the unit file, please raise it freely.

Thanks!

Answer 1

There's a difference between the "daemon" failing and the "service" being in a failed state.

The daemon failing simply means if the program exited somehow other than returning 0 as an exit code. That's what Restart=on-failure is referring to i.e. "Restart the daemon if it fails" Also, you can restart on other conditions see the table under Restart= which also explains what "on-failure" exactly means.

According to your service file, there is no definition of "service" failure, and I think the default definition is basically "the daemon exited somehow other than returning 0 as an exit code AND it is not currently running", but your service file has the daemon restarted immediately. So it probably does enter the "failed" state, but so briefly you don't notice it. You can check the log with journalctl -u foo.service.

Also, you should check to see if the PID of the main process is still the same as it was +30 seconds ago. If the PID is different, that means systemd is killing the daemon, because it never called sd_notify and then systemd is restarting the daemon very quickly because your service file says to restart it on failure, and doesn't tell it to wait between restarts (see below).

If you want your service to have a definition of failure you want to use something like:

StartLimitInterval=5min
StartLimitBurst=4

Those options are documented in depth here, however what these two basically mean in this example is "Start the daemon no more than 4 times in a 5 minute interval. If it is started, and then fails, that many times in that interval, the service is considered to be in a failed state". If that happens, then the StartLimitAction= is taken, by default that is none.

Another configurable value is RestartSec=. From the documentation:

Configures the time to sleep before restarting a service (as configured with Restart=). Takes a unit-less value in seconds, or a time span value such as "5min >20s". Defaults to 100ms.

Which basically means that systemd must wait at least RestartSec= before starting the daemon again.

So you can combine all of these options, but let's say you use these values:

StartLimitInterval=5min
StartLimitBurst=4
RestartSec=2min

Then the service could never enter the failed state, since the definition of failure is:

If in a given 5 minute interval, the service has been started, and then failed 4 times.

But systemd won't re-start the service for two minutes after the last failure, so it would never reach the StartLimitBurst inside of the StartLimitInterval.