0

I have a Raspberry Pi running Raspbian GNU/Linux 10 (buster), which has an onboard watchdog timer. I have configured this with a test script that looks for /tmp/force-reboot.txt, and if that file exists, the test script fails, and will force a reset if it stays present for 60 seconds. This works great in casual testing -- I type touch /tmp/force-reboot.txt and the system reboots 60 seconds later.

The problem is that even when I use this, I still get intermittent hangs on reboot. I am trying to protect against such a hang on reboot with touch /tmp/force-reboot.txt; /sbin/reboot This does not stop hangs on reboot because the intermittent hang happens after the system shuts down the watchdog service. Below is an excerpt of /var/log/syslog from when it hung last night.

I see that /etc/rc6.d contains a symlink to K01watchdog -> ../init.d/watchdog, which is what causes this shutdown of the watchdog on reboot. But even though the watchdog is the last entry in that directory (alphabetically) and so in theory executes last, something still hangs the system.

Should I delete the K01watchdog symlink in /etc/rc6.d to prevent it from stopping the service on shutdown? This seems like a messy solution, so is there a cleaner way? Ideally I would find the cause of the hang, but after many weeks, I have been unable to do so -- finding the cause of intermittent problems is very difficult.

Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: watchdog.service: Control process exited, code=exited, status=1/FAILURE
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: watchdog.service: Failed with result 'exit-code'.
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: Stopped watchdog daemon.
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: watchdog.service: Triggering OnFailure= dependencies.
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: Requested transaction contradicts existing jobs: Transaction for wd_keepalive.service/start is destructive (systemd-fsck-root.service has 'stop' job queued, but 'start' is included in transaction).
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: watchdog.service: Failed to enqueue OnFailure= job, ignoring: Transaction for wd_keepalive.service/start is destructive (systemd-fsck-root.service has 'stop' job queued, but 'start' is included in transaction).
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: alsa-state.service: Succeeded.
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: Stopped Manage Sound Card State (restore and store).
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: udisks2.service: Succeeded.
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: Stopped Disk Manager.
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: bluetooth.service: Succeeded.
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: Stopped Bluetooth service.
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: Received SIGRTMIN+20 from PID 13920 (plymouthd).
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: user@1000.service: Succeeded.
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: Stopped User Manager for UID 1000.
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: user@0.service: Succeeded.
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: Stopped User Manager for UID 0.
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: Stopping User Runtime Directory /run/user/0...
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: Stopping User Runtime Directory /run/user/1000...
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: Stopped target Multi-User System.
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: rc-local.service: Succeeded.
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: Stopped /etc/rc.local Compatibility.
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: Stopping Configure Bluetooth Modems connected by UART...
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: gldriver-test.service: Succeeded.
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: Stopped Check for v3d driver.
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: Stopping Login Service...
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: Stopped target Login Prompts.
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: Stopping dphys-swapfile - set up, mount/unmount, and delete a swap file...
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: Stopping LSB: A service to forward local ports to the internet...
Oct 11 04:00:01 Rsrvd-CTLR-EXIT systemd[1]: Stopping triggerhappy global hotkey daemon...

-- no further logs until power cycle --

EDIT: I tried deleting /etc/rc6.d/K01watchdog, (on a different Pi with the same configuraton) and commanded reboots still seem to work, although I do see these logs during the shutdown. Not sure if this is a problem:

Oct 17 10:40:05 Capitol-Hill-CTLR-ENTRY systemd[1]: alsa-state.service: Succeeded.
Oct 17 10:40:05 Capitol-Hill-CTLR-ENTRY systemd[1]: Stopped Manage Sound Card State (restore and store).
Oct 17 10:40:05 Capitol-Hill-CTLR-ENTRY systemd[1]: watchdog.service: Control process exited, code=exited, status=1/FAILURE
Oct 17 10:40:05 Capitol-Hill-CTLR-ENTRY systemd[1]: watchdog.service: Failed with result 'exit-code'.
Oct 17 10:40:05 Capitol-Hill-CTLR-ENTRY systemd[1]: Stopped watchdog daemon.
Oct 17 10:40:05 Capitol-Hill-CTLR-ENTRY systemd[1]: watchdog.service: Triggering OnFailure= dependencies.
Oct 17 10:40:05 Capitol-Hill-CTLR-ENTRY systemd[1]: Requested transaction contradicts existing jobs: Transaction for wd_keepalive.service/start is destructive (local-fs.target has 'stop' job queued, but 'start' is included in transaction).
Oct 17 10:40:05 Capitol-Hill-CTLR-ENTRY systemd[1]: watchdog.service: Failed to enqueue OnFailure= job, ignoring: Transaction for wd_keepalive.service/start is destructive (local-fs.target has 'stop' job queued, but 'start' is included in transaction).
Oct 17 10:40:05 Capitol-Hill-CTLR-ENTRY systemd[1]: Stopped target Multi-User System.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.