0

I am running AlmaLinux 9, and it has been running stable for some time. This morning I received email messages that a systemd service had restarted on its own overnight. Strange, since this service is NOT set to autorestart, just start with system. The host is on a UPS so there shouldn't have been any strange power issues. I checked the boot log, and the "last" command shows:

reboot   system boot  5.14.0-284.25.1. Mon Nov 20 02:15   still running
reboot   system boot  5.14.0-284.25.1. Sun Oct 29 15:24   still running
reboot   system boot  5.14.0-284.25.1. Thu Oct 26 10:57 - 11:07 (2+00:09)
reboot   system boot  5.14.0-284.25.1. Wed Oct 25 17:13 - 17:16  (00:03)
reboot   system boot  5.14.0-284.25.1. Fri Oct 20 11:56 - 17:10 (5+05:13)
reboot   system boot  5.14.0-284.25.1. Wed Oct 18 20:25 - 11:51  (15:26)
reboot   system boot  5.14.0-284.25.1. Tue Oct 17 04:52 - 11:51 (2+06:59)
reboot   system boot  5.14.0-284.25.1. Fri Sep 29 15:45 - 11:51 (19+20:06)
reboot   system boot  5.14.0-284.25.1. Sat Sep 23 09:09 - 14:56 (6+05:46)
reboot   system boot  5.14.0-284.25.1. Tue Sep 19 20:14 - 09:09 (3+12:55)
reboot   system boot  5.14.0-284.25.1. Tue Sep 19 19:21 - 19:57  (00:35)
reboot   system boot  5.14.0-284.25.1. Sun Sep 10 20:23 - 18:01 (8+21:37)
reboot   system boot  5.14.0-284.25.1. Sun Sep 10 16:18 - 20:23  (04:04)
reboot   system boot  5.14.0-284.25.1. Fri Sep  8 20:53 - 19:56  (23:03)
reboot   system boot  5.14.0-284.25.1. Fri Sep  8 20:43 - 20:52  (00:08)
reboot   system boot  5.14.0-284.25.1. Fri Sep  8 20:27 - 20:43  (00:15)
reboot   system boot  5.14.0-284.25.1. Fri Sep  8 19:10 - 20:26  (01:16)
reboot   system boot  5.14.0-284.25.1. Fri Sep  8 16:25 - 18:52  (02:26)
reboot   system boot  5.14.0-284.11.1. Fri Sep  8 19:50 - 16:24  (-3:25)

Which doesn't make sense...how can the top two lines both be "still running"? Or am I misinterpreting what this means?

I check uptime --pretty and it says:

up 8 hours, 49 minutes

Yet my /var/log/messages file does not show any system restart related messages around 9 hours ago...just regular logging.

What is going on? Uptime says my host rebooted this morning, "last" says the Oct 29 start is still running (but so is a Nov 20 / today start), and the system logs shows no indication of a restart in the last 24 hours. This makes no sense

2 Answers 2

0

Did you also check the previous version of /var/log/messages?

Monday morning at about 04:00 would be a pretty typical time for running automatic weekly log rotations. If that happened on your system, the restart messages may simply have already been rotated to the /var/log/messages.1[.gz] or whatever naming scheme is used by AlmaLinux's log rotation.

If the logs of the restart(s?) are really gone, then I would worry about a possible hacker/malware attack. A reboot with logs (possibly inexpertly/incompletely) deleted afterwards might indicate that a rootkit has been installed to your system, to hide the processes run by the intruder and to make sure the intruder will be able to regain access to the system even if the original vulnerability that allowed the intruder to get in is fixed.

I would also recommend stress testing the UPS battery if it is not quite new, by disconnecting the actual computer from the UPS, and making the UPS power some "harmless" load of roughly equivalent wattage instead, like a desk lamp or a fan for example, and then seeing how long the UPS can actually provide power after disconnecting incoming power from the wall.

If your UPS's batteries are nearly dead, the system may have suffered a sudden loss of power, then restarted and lost power again in mid-startup, which may have caused some anomalies and data losses in the respective logs if the second power loss happened at the exactly wrong moment.

(My UPS recently started notifying me that its batteries needed replacing. I procrastinated... and then there was a very brief power outage, and the UPS failed after just a few seconds, as the batteries were indeed totally dead. I now have learned my lesson, and also have the new batteries installed.)

1
  • I checked the rotated logs and there is no indication of a system restart - but all logs are present. (And system is not exposed to internet, so not likely to have been hacked). My UPS log doesn't show any restart, and other hosts on the UPS have not restarted.
    – TSG
    Nov 20, 2023 at 17:16
0

To answer the title of you question, here are things you can try:

  1. date -d@$(cat /proc/uptime | cut -f 1 -d " ") -u +%H:%M:%S

This is just a fancy way of translating the output from /proc/uptime to a nicer output (and remove the "idle time").

  1. ps -p 1 -o lstart=

This doesn't show you when the system started but rather when the init system started. If you're not in a container this is equivalent to when the system last rebooted. You can also run date --date="$(cat /proc/uptime | cut -f 1 -d " ") seconds ago" to see if these options match up.

  1. dmesg -H | head -n 1

This should show you the time dmesg started writing to log. This is a binary log so it is more reliable than just a text file, but not infallible.

2
  • The latter command reports "Mon Nov 20 01:15:42 AM EST 2023" - so init restarted this morning. But..why does LAST think 2 system reboots are still running at once? (Does this make sense)...and yet my system log does not indicate any reboot at 1:15am
    – TSG
    Nov 20, 2023 at 17:23
  • I'd read a bit more in dmesg. Maybe even just dmesg -H. If it looks like the system rebooted then it is very likely that it did. dmesg just prints the kernels cyclic buffer, as such it is unlikely to lie. I'm unfamiliar with last so take this a whole box of salt, but I'd guess what you're seeing there was a crash or something else that prevented the user from exiting in an organized manner. You can check with less /etc/passwd if you have two root users if you suspect something like that.
    – Uberhumus
    Nov 20, 2023 at 17:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .