0

I have systemd running with a watchdog, and a service that uses a systemd watchdog to restart the service when it locks up. If the service locks up repeatedly then systemd reboots the machine.

How can I log when systemd restarts the service and when the machine is rebooted by systemd?

The OS is ARCH linux and the systemd version is 218.

Jounrnalctl is empty for that day after a watchdog reboot.

I don't know what the journal contains if only the service is restarted. Whats the most effective way of testing an service hanging?

/var/log/journal exists and will store information on a normal reboot, but doesn't have any logs for the day on a watchdog reboot.

Reading Debugging lock-up - systemd loses my logs which seems to be a similiar problem. Uncommenting #SyncIntervalSec=5m in my journald.conf gives me logs of the reboot, but no clue as to what triggers it.

In this case it was a forkbomb causing systemd to reboot the whole machine.

Apr 11 20:54:02 buspi systemd[1]: Stopping Sound Card.   <--- restarting
Apr 11 20:54:01 buspi anacron[17809]: Job `cron.daily' started <--- seems ok

Unit File

[Unit]
Description= Alight
Wants=network.target

[Service]
Type=notify
ExecStart=/usr/bin/alight 
ExecStop=/usr/bin/alight-stop
Restart=always
WatchdogSec=30s
StartLimitInterval=5min
StartLimitBurst=4
StartLimitAction=reboot-force

[Install]
WantedBy=multi-user.target

system.conf

[Manager]
#LogLevel=info
#LogTarget=journal-or-kmsg
#LogColor=yes
#LogLocation=no
#DumpCore=yes
#CrashShell=no
#ShowStatus=yes
#CrashChVT=1
#CPUAffinity=1 2
#JoinControllers=cpu,cpuacct net_cls,net_prio
RuntimeWatchdogSec=20
ShutdownWatchdogSec=3min
#CapabilityBoundingSet=
#SystemCallArchitectures=
#TimerSlackNSec=
#DefaultTimerAccuracySec=1min
#DefaultStandardOutput=journal
#DefaultStandardError=inherit
#DefaultTimeoutStartSec=90s
#DefaultTimeoutStopSec=90s
#DefaultRestartSec=100ms
#DefaultStartLimitInterval=10s
#DefaultStartLimitBurst=5
#DefaultEnvironment=
#DefaultCPUAccounting=no
#DefaultBlockIOAccounting=no
#DefaultMemoryAccounting=no
#DefaultLimitCPU=
#DefaultLimitFSIZE=
#DefaultLimitDATA=
#DefaultLimitSTACK=
#DefaultLimitCORE=
#DefaultLimitRSS=
#DefaultLimitNOFILE=
#DefaultLimitAS=
#DefaultLimitNPROC=
#DefaultLimitMEMLOCK=
#DefaultLimitLOCKS=
#DefaultLimitSIGPENDING=
#DefaultLimitMSGQUEUE=
#DefaultLimitNICE=
#DefaultLimitRTPRIO=
#DefaultLimitRTTIME=
  • 1
    It should already do so. What's in your journal before the machine is rebooted? – maxf Aug 27 '16 at 10:23
  • What OS and version are you using? – Mark Stosberg Aug 29 '16 at 19:44
1

If /var/log/journal does not exist, by default the systemd journal will be stored in-memory and thus lost on reboot.

Review the official docs for systemd's journal, particularly the list of config files at the top that you might wish to review and the Storage= option.

It could be that the logging is happening but is being lost during reboot before you can review due to your Storage= configuration for the systemd journal.

  • /var/log/journal/ exists and Storage=persistent. Logs are still lost on watchdog reboot. – Robert Pringle Aug 30 '16 at 9:29
  • That sounds like a bug to me. I would expect flushing to disk to be happening frequently enough that you would have /most/ logs, even if the system crashed suddenly and the final few logs were lost. – Mark Stosberg Aug 31 '16 at 18:01
  • 1
    Reading Debugging lock-up - systemd loses my logs which seems to be a similiar problem. Uncommenting #SyncIntervalSec=5m in my journald.conf gives me logs of the reboot – Robert Pringle Aug 31 '16 at 20:37
  • Since you found the answer, you should turn your comment into an answer and then mark it as "accepted". Also, link to the other post you are referring to. – Mark Stosberg Sep 1 '16 at 13:38
  • So I have logs during the reboot, but no indication that the reboot was triggered by the watched. I've edited the original question to reflect this. – Robert Pringle Sep 1 '16 at 13:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.