I am running Arch-ARM on RPi3. I have noticed when system crashes I cannot find any related crash log in journal logs.
Arch Linux ARM on RPi3: Linux 4.4.37-1-ARCH #1 SMP armv7l GNU/Linux
Systemd: systemd 232
/etc/systemd/journald.conf:
[Journal]
Storage=persistent
Compress=yes
#Seal=yes
#SplitMode=uid
SyncIntervalSec=1
#RateLimitIntervalSec=30s
#RateLimitBurst=1000
SystemMaxUse=1.5G
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
MaxRetentionSec=1month
MaxFileSec=3hour
#ForwardToSyslog=no
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
Recent crash log:
Dec 29 03:43:48 sudo[21861]: my_user : TTY=unknown ; PWD=/opt/my_app/repo/src ; USER=root ; COMMAND=/usr/sbin/hciconfig hci0 reset
Dec 29 03:43:48 sudo[21861]: pam_unix(sudo:session): session opened for user root by (uid=0)
Dec 29 03:43:48 sudo[21861]: pam_unix(sudo:session): session closed for user root
Dec 29 03:43:48 my_app.py[17773]: trying to connect to XX:XX:XX:XX:XX:XX
Dec 29 03:43:48 systemd-udevd[21865]: Process '/bin/hciconfig hci0:64 up' failed with exit code 1.
Dec 29 03:43:51 my_app.py[17773]: connection successful :)
-- Reboot --
Jan 03 16:31:25 systemd[1]: Time has been changed
Jan 03 16:31:26 dhcpcd[470]: forked to background, child pid 587
Jan 03 16:31:25 systemd-timesyncd[360]: Synchronized to time server 206.108.0.133:123 (2.arch.pool.ntp.org).
Jan 03 16:31:25 systemd[1]: Starting Update man-db cache...
Jan 03 16:31:25 systemd[1]: Starting Rotate log files...
Jan 03 16:31:25 systemd[1]: Started Verify integrity of password and group files.
Jan 03 16:31:25 systemd[1]: ssh-tunnel.service: Service hold-off time over, scheduling restart.
Looks like that somehow journald is failing to sync logs when a crash happens.
- Is this a known behaviour?
- Is there a workaround for this?
Also I am curious to know if the following claim from Arch Linux wiki is still valid:
Since the syslog component of systemd, journald, does not flush its logs to disk during normal operation, these logs will be gone when the machine is shut down abnormally (power loss, kernel lock-ups, ...). In the case of kernel lock-ups, it is pretty important to have some kernel logs for debugging. Until journald gains a configuration option for flushing kernel logs, rsyslog can be used in conjunction with journald.
related bug report (old): Bug 61411 - All logs since last boot gone after crash/hard reboot
similar issue (old): Debugging lock-up - systemd loses my logs
journalctl, which would have an additional field.