0

I have systemd running with a watchdog, and a service that uses a systemd watchdog to restart the service when it locks up. If the service locks up repeatedly then systemd reboots the machine.

How can I log when systemd restarts the service and when the machine is rebooted by systemd?

The OS is ARCH linux and the systemd version is 218.

Jounrnalctl is empty for that day after a watchdog reboot.

I don't know what the journal contains if only the service is restarted. Whats the most effective way of testing an service hanging?

/var/log/journal exists and will store information on a normal reboot, but doesn't have any logs for the day on a watchdog reboot.

Reading Debugging lock-up - systemd loses my logs which seems to be a similiar problem. Uncommenting #SyncIntervalSec=5m in my journald.conf gives me logs of the reboot, but no clue as to what triggers it.

In this case it was a forkbomb causing systemd to reboot the whole machine.

Apr 11 20:54:02 buspi systemd[1]: Stopping Sound Card.   <--- restarting
Apr 11 20:54:01 buspi anacron[17809]: Job `cron.daily' started <--- seems ok

Unit File

[Unit]
Description= Alight
Wants=network.target

[Service]
Type=notify
ExecStart=/usr/bin/alight 
ExecStop=/usr/bin/alight-stop
Restart=always
WatchdogSec=30s
StartLimitInterval=5min
StartLimitBurst=4
StartLimitAction=reboot-force

[Install]
WantedBy=multi-user.target

system.conf

[Manager]
#LogLevel=info
#LogTarget=journal-or-kmsg
#LogColor=yes
#LogLocation=no
#DumpCore=yes
#CrashShell=no
#ShowStatus=yes
#CrashChVT=1
#CPUAffinity=1 2
#JoinControllers=cpu,cpuacct net_cls,net_prio
RuntimeWatchdogSec=20
ShutdownWatchdogSec=3min
#CapabilityBoundingSet=
#SystemCallArchitectures=
#TimerSlackNSec=
#DefaultTimerAccuracySec=1min
#DefaultStandardOutput=journal
#DefaultStandardError=inherit
#DefaultTimeoutStartSec=90s
#DefaultTimeoutStopSec=90s
#DefaultRestartSec=100ms
#DefaultStartLimitInterval=10s
#DefaultStartLimitBurst=5
#DefaultEnvironment=
#DefaultCPUAccounting=no
#DefaultBlockIOAccounting=no
#DefaultMemoryAccounting=no
#DefaultLimitCPU=
#DefaultLimitFSIZE=
#DefaultLimitDATA=
#DefaultLimitSTACK=
#DefaultLimitCORE=
#DefaultLimitRSS=
#DefaultLimitNOFILE=
#DefaultLimitAS=
#DefaultLimitNPROC=
#DefaultLimitMEMLOCK=
#DefaultLimitLOCKS=
#DefaultLimitSIGPENDING=
#DefaultLimitMSGQUEUE=
#DefaultLimitNICE=
#DefaultLimitRTPRIO=
#DefaultLimitRTTIME=
Improve this question
2
  • 1
    It should already do so. What's in your journal before the machine is rebooted?
    – maxf
    Commented Aug 27, 2016 at 10:23
  • What OS and version are you using?
    – Mark Stosberg
    Commented Aug 29, 2016 at 19:44

1 Answer 1

Reset to default
1

If /var/log/journal does not exist, by default the systemd journal will be stored in-memory and thus lost on reboot.

Review the official docs for systemd's journal, particularly the list of config files at the top that you might wish to review and the Storage= option.

It could be that the logging is happening but is being lost during reboot before you can review due to your Storage= configuration for the systemd journal.

Improve this answer
7
  • /var/log/journal/ exists and Storage=persistent. Logs are still lost on watchdog reboot.
    – Robert Pringle
    Commented Aug 30, 2016 at 9:29
  • That sounds like a bug to me. I would expect flushing to disk to be happening frequently enough that you would have /most/ logs, even if the system crashed suddenly and the final few logs were lost.
    – Mark Stosberg
    Commented Aug 31, 2016 at 18:01
  • 1
    Reading Debugging lock-up - systemd loses my logs which seems to be a similiar problem. Uncommenting #SyncIntervalSec=5m in my journald.conf gives me logs of the reboot
    – Robert Pringle
    Commented Aug 31, 2016 at 20:37
  • Since you found the answer, you should turn your comment into an answer and then mark it as "accepted". Also, link to the other post you are referring to.
    – Mark Stosberg
    Commented Sep 1, 2016 at 13:38
  • So I have logs during the reboot, but no indication that the reboot was triggered by the watched. I've edited the original question to reflect this.
    – Robert Pringle
    Commented Sep 1, 2016 at 13:44

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .