1

I have been investigating to fix what appears to be a problem with systemd watchdog but with no success. I have a bunch of services that are of Type=notify. If the system time is set backwards, every service is killed by systemd because it thinks the watchdog timeout was hit. Here is the debug log from systemd

systemd[1]: xxx:service: Unit entered failed state.
systemd[1]: xxx:service: Failed with result 'watchdog'.

Is there a way to tell Systemd not to restart services when the system time is changed manually?

Shouldn't the watchdog depend on the CLOCK_MONOTONIC? Is this a known issue in Systemd?

  • 1
    I'd suggest using NTP to keep the time adjusted rather than setting the time manually. NTP wouldn't jump the time backwards but slow down the clock until it has caught up with real time (provided that the clock was not too far wrong to start with). – Kusalananda May 14 '18 at 8:13
  • 1
    Post the version of your systemd! Including where it came from, i.e. to see if it includes custom patches. – sourcejedi May 15 '18 at 18:42
  • 1
    It should. On a quick check, it appears to be written with reference to CLOCK_MONOTONIC for at least 4 years. github.com/systemd/systemd/blame/master/src/core/service.c#L224 Maybe there's a weird bug though. – sourcejedi May 15 '18 at 18:57
0

Is there a way to tell Systemd not to restart services when the system time is changed manually?

No. According to the docs in man systemd.service for WatchDogSec=, there is no promise that systemd will be tracking this using a monotonic timer, so it working as advertised.

If you think the behavior should be changed, you could discuss with the systemd developers or open an issue:

https://github.com/systemd/systemd/issues?utf8=%E2%9C%93&q=WatchDogSec%3D

|improve this answer|||||
  • 1
    It's clearly advertised as a watchdog timer, and it is clearly not correct to implement a watchdog timer using wall-clock time. – sourcejedi May 15 '18 at 18:44
  • @sourcejedi If you feel strongly about that, file a bug report. – Mark Stosberg May 16 '18 at 13:40
  • If the watchdog timer was based on wall-clock time, GNOME (either X or gnome-shell under Wayland) would crash after suspending your computer for more than 3 minutes. (logind has a watchdog timer, and killing/restarting logind causes X, and all the Wayland compositors I've seen, to die). It needs to use a monotonic clock, of the sort that don't count time spent in suspend. See also "CLOCK_MONOTONIC change reverted". – sourcejedi May 16 '18 at 18:20
  • @sourcejedi If you feel strongly the behavior is wrong, file a bug report. – Mark Stosberg May 16 '18 at 18:28
  • Waiting on OP (or anyone else) to specify an affected systemd version. I don't want to spend my time trying to reproduce a weird bug like this without knowing which version. – sourcejedi May 16 '18 at 18:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.