E.g. I'm seeing this in
Mar 01 23:12:34 hostname shutdown: shutting down for system halt
Is there a way to find out what caused the shutdown? E.g. was it run from console, or someone hit power button, etc.?
Only root privileged programs can gracefully shutdown a system. So when a system shuts down in a normal way, it is either a user with root privileges or an acpi script. In both cases you can find out by checking the logs. An acpi shutdown can be caused by power button press, overheating or low battery (laptop). I forgot the third reason, UPS software when power supply fails, which will send an alert anyway.
Recently I had a system that started repeatedly to power off ungracefully, turned out that it was overheating and the mobo was configured to just power off early. The system didn't have a chance to save logs, but fortunately monitoring the system's temperature showed it was starting to increase just before powering off.
So if it is a normal shutdown it will be logged, if it is an intrusion... good luck, and if it is a cold shutdown your best chance to know is to control and monitor its environment.
Try the following commands:
Display list of last reboot entries:
last reboot | less
Display list of last shutdown entries:
last -x | less
or more precisely:
last -x | grep shutdown | less
You won't know who did it however. If you want to know who did it, you will need to add a bit of code which means you'll know next time.
I've found this resource online. It might be useful to you:
You have to examine 2 things:
Use these 2 commands and keep reading for more information.
last -x | head | tac grep -iv ': starting\|kernel: .*: Power Button\|watching system buttons\|Stopped Cleaning Up\|Started Crash recovery kernel' \ /var/log/messages /var/log/syslog /var/log/apcupsd* \ | grep -iw 'recover[a-z]*\|power[a-z]*\|shut[a-z ]*down\|rsyslogd\|ups'
Run this command* and compare the output to the examples below:
last -x | head | tac
A normal shutdown and power-up looks like this (note that you have a shutdown event and then a system boot event):
runlevel (to lvl 0) 2.6.32- Sat Mar 17 08:48 - 08:51 (00:02) shutdown system down ... <-- first the system shuts down reboot system boot ... <-- afterwards the system boots runlevel (to lvl 3)
In some cases you may see this (note that there is no line about the shutdown but the system was at runlevel 0 which is the "halt state"):
runlevel (to lvl 0) ... <-- first the system shuts down (init level 0) reboot system boot ... <-- afterwards the system boots runlevel (to lvl 2) 2.6.24-... Fri Aug 10 15:58 - 15:32 (2+23:34)
An unexpected shutdown from power loss looks like this (note that you have a system boot event without a prior system shutdown event):
runlevel (to lvl 3) ... <-- the system was running since this momemnt reboot system boot ... <-- then we've a boot WITHOUT a prior shutdown runlevel (to lvl 3) 3.10.0-693.21.1. Sun Jun 17 15:40 - 09:51 (18:11)
A bash command to filter the most interesting log messages is this:
grep -iv ': starting\|kernel: .*: Power Button\|watching system buttons\|Stopped Cleaning Up\|Started Crash recovery kernel' \ /var/log/messages /var/log/syslog /var/log/apcupsd* \ | grep -iw 'recover[a-z]*\|power[a-z]*\|shut[a-z ]*down\|rsyslogd\|ups'
When an unexpected power off or hardware failure occurs the filesystems will not be properly unmounted so in the next boot you may get logs like this:
EXT4-fs ... INFO: recovery required ... Starting XFS recovery filesystem ... systemd-fsck: ... recovering journal systemd-journald: File /var/log/journal/.../system.journal corrupted or uncleanly shut down, renaming and replacing.
When the system powers off because user pressed the power button you get logs like this:
systemd-logind: Power key pressed. systemd-logind: Powering Off... systemd-logind: System is powering down.
Only when the system shuts down orderly you get logs like this:
rsyslogd: ... exiting on signal 15
When the system shuts down due to overheating you get logs like this:
critical temperature reached...,shutting down
If you have a UPS and running a daemon to monitor power and shutdown you should obviously check its logs (NUT logs on /var/log/messages but apcupsd logs on /var/log/apcupsd*)
*: Here's the description of
last from its man page:
last [...] prints information about connect times of users. Records are printed from most recent to least recent. [...] The special users reboot and shutdown log in when the system reboots or (surprise) shuts down.
head to keep the latest 10 events and we use
tac to invert the ordering so that we don't get confused by the fact that last prints from most recent to least recent event.
Some possible log files to explore: (found a Ubuntu system, but I would hope that they're present on most Linux/Unix systems)
/var/log/debug /var/log/syslog (will be pretty full and may be harder to browse) /var/log/user.log /var/log/kern.log /var/log/boot
Again, these log files are present on a Ubuntu system, so filenames may be different. The
tail command is your friend.
I had a similar need on a Debian 7.8 and observe that basically there's no clear and explicit message in log, which is a little surprising.
/var/log would tell the time the machine was shut down, show proper daemons shutdown, etc, but not the initial reason.
shutdown: shutting down for system halt
The other solutions mentioned (
last -x) did not help much.
/etc/acpi/powerbtn-acpi-support.sh which includes:
if [ -x /etc/acpi/powerbtn.sh ] ; then # Compatibility with old config script from acpid package /etc/acpi/powerbtn.sh elif [ -x /etc/acpi/powerbtn.sh.dpkg-bak ] ; then # Compatibility with old config script from acpid package # which is still around because it was changed by the admin /etc/acpi/powerbtn.sh.dpkg-bak else # Normal handling. /sbin/shutdown -h -P now "Power button pressed" fi
Notice that an explicit text is given as parameter of the
shutdown command. I would expect that string to be logged automatically by the shutdown program.
Anyway, to get an explicit message I put the text below (as root) in a newly created
/etc/acpi/powerbtn.sh made executable with
chmod a+x /etc/acpi/powerbtn.sh
#!/bin/sh logger in /etc/acpi/powerbtn.sh, presumably "Power button pressed" /sbin/shutdown -h -P now "Power button pressed"
Doing it this way will probably make a longer lasting change than modifying
/etc/acpi/powerbtn-acpi-support.sh. The latter option would probably lose its effect on next upgrade of package
Notice than Ubuntu 14.04 does it differently (
/etc/acpi/powerbtn.sh already exists with different content from
acpid package). Also, Debian 8 probably does it differently. Feel free to offer variants.
And now when the power button is pressed, a line like below appears in
logger: in /etc/acpi/powerbtn.sh, presumably Power button pressed
Now that's an explicit message in the log.
I have just a clumsy idea, but maybe it works for you:
enter the command
last and check out the login informations for all of the users. then, filter th users with the permission required for
halt that had been logged in at that moment. then check out their
.bash_history file to see if they have entered halt or not.
Just chip in that on my KVM VM (where I wondered whether a host reboot did a clean shutdown of guests), I found what I needed in
/var/log/auth.log (in addition to
last -x shutdown showing the same). There these lines showed up:
Sep 3 23:56:31 Web systemd-logind: Power key pressed. Sep 3 23:56:31 Web systemd-logind: Powering Off... Sep 3 23:56:31 Web systemd-logind: System is powering down. Sep 3 23:55:45 Web systemd-logind: New seat seat0. Sep 3 23:55:45 Web systemd-logind: Watching system buttons on /dev/input/event0 (Power Button) Sep 3 23:55:54 Web sshd: Server listening on 0.0.0.0 port 22. Sep 3 23:55:54 Web sshd: Server listening on :: port 22.
last -x shows these lines, notice that they're being printed in most-recent-first order (i.e. read the last line first, and then go up), but due to the clock reset (23:56 before boot, 23:55 after) also evident in the previous lines, the order seems a little bewildering:
runlevel (to lvl 2) 3.13.0-129-gener Sun Sep 3 23:55 - 22:04 (22:08) reboot system boot 3.13.0-129-gener Sun Sep 3 23:55 - 22:04 (22:08) shutdown system down 3.13.0-123-gener Sun Sep 3 23:56 - 23:55 (00:00) runlevel (to lvl 0) 3.13.0-123-gener Sun Sep 3 23:56 - 23:56 (00:00)
For my part, checking that guests get cleanly shut down when host is booted, I could also just log into (ssh) one of the guests, and stay there when I boot the host, getting these lines in the terminal:
root@Web:~# Broadcast message from root@Web (unknown) at 22:25 ... The system is going down for power off NOW! Connection to web closed by remote host. Connection to web closed.