Does systemd use a different timeout setting for stopping a running daemon (e.g.,
rsyslog) when rebooting the system (e.g., by running
reboot) vs when you simply restart the daemon (e.g.,
systemctl restart rsyslog)?
I've checked the systemd.service page, but I didn't spot it. Instead I only found the
TimeoutStartSecoptions. I've set the
TimeoutStopSec option, but it appears that systemd may be killing the daemon before it has a chance to safely save its state and terminate cleanly.
As @sourcejedi suggested (thanks), I should emphasize that this is not a desktop installation where rsyslog is running, but an Ubuntu 16.04 server install of rsyslog that receives messages from client nodes and may still be holding many messages in memory when asked to terminate by systemd.
I attempted to help work around some corrupt disk queue issues by increasing the value for the
TimeoutStopSec option from 90 seconds to 240 seconds, but I still observed this message multiple times in the related log file:
rsyslogd: queue 'strm 0x26b4800', file '/var/spool/rsyslog/q_ForwardToNode2.00000003' opened for non-append write, but already contains 983505 bytes [v8.29.0 try http://www.rsyslog.com/e/0 ]
The idea was that system might be impatient and was killing rsyslog while it was still saving content to disk.
I attempted to work around another issue by forcing systemd to wait on an active network connection before attempting to start rsyslog. I've included the contents of both systemd
Drop-Ins that I am using below for reference in case it adds helpful context to this entry.
cat /etc/systemd/system/rsyslog.service.d/*.conf | grep -Ev '#|^$'
Attempt to work around github #1656
[Unit] Documentation=https://internal/wiki/url/here After=network.target Wants=nework.target
Attempt to work around github #1704
[Unit] Documentation=https://internal/wiki/url/here [Service] TimeoutStopSec=240
Thank you for reading this.