0

My Supermicro blade server shut itself down with the following logs. What could be the reason? (AFIK no one has touched it!)

  • CPU usage was about 50% in the few hours before the incident.
  • This issue had happened a few time in the past few weeks, all were when the was a similar load on the system.
  • There is no entry of high temperature (or anything else) in the BIOS event log.
  • There is no event registered in the IPMI log
  • CentOS Linux release 7.5.1804, kernel 3.10.0-862.6.3.el7.x86_64
09:04:30 hostname ***some event***
09:30:52 hostname systemd-logind: Power key pressed.
09:30:52 hostname systemd-logind: Power key pressed.
09:30:52 hostname systemd-logind: Powering Off...
09:30:52 hostname systemd-logind: System is powering down.
09:30:52 hostname systemd-logind: Powering Off...
09:30:52 hostname systemd-logind: System is powering down.
09:30:52 hostname systemd: Stopping Authorization Manager...
09:30:52 hostname systemd: Stopping Authorization Manager...
  • 5
    Did someone press the power key? – Kusalananda Jul 9 '18 at 16:01
  • That is not a title, please fix. (I will continue reading when you have done this.) – ctrl-alt-delor Jul 9 '18 at 16:07
  • Does the blade have a ILO or system management with logs that are independent of the OS? – thebtm Jul 9 '18 at 16:23
  • The post is not explicit when does it happen, and how many times. – Rui F Ribeiro Jul 9 '18 at 16:34
  • A faulty power supply is one of the simplest explanations. – Satō Katsura Jul 9 '18 at 16:40
4

The most obvious answer is, of course, that someone pushed the power button. Possibly accidentally — e.g., if the power button is unfortunately sensitive and someone bumped it slightly while walking by. This can also happen with a failing power button (e.g., springs have worn out). Accidental power button presses can be helped by fashioning some sort of Molly guard to install over the power button.

Asking for a graceful shutdown from, e.g., IPMI will often also register as power button pressed.

Another possibility is (electrical) noise making the system think the power button has been pressed. Make sure the wires to the power button are firmly seated on the board (and the switch, if not soldered) and check cable routing to make sure the wires are away from anything with large/variable power consumption.

If nothing else works, you can edit /etc/systemd/logind.conf and change HandlePowerKey to ignore (then restart systemd-logind). That should stop the system from shutting down, but it does mean that you won't be able to use the power button to initiate a clean shutdown (the hold-for-4-seconds unclean shutdown, if any, can't be disabled via systemd as it's done by the system firmware).

  • Yeah, faulty button or someone is pressing it, cat or cleaning lady. – Rui F Ribeiro Jul 9 '18 at 18:18
  • Thank you derobert. I will start testing from the the last item, setting HandlePowerKey to ignore. If that solves the issue, what would it mean!? – user1507435 Jul 10 '18 at 15:49
  • @user1507435 doesn't really tell you much, that's just configuring the system to ignore the power button press its detecting. – derobert Jul 10 '18 at 15:51
  • After setting HandlePowerKey to ignore, the server shutdown, after a week, without any error in the syslog or any event in the ipmi log. The poweroff button is very hard to reach, so I can safely skip trying a Molly guard. If this shutdown is is because of a CPU getting too hot, should I expect to see an event logged somewhere? – user1507435 Jul 17 '18 at 13:49
  • @user1507435 was it a clean shutdown? If overheating caused a clean shutdown, I'd expect logs about it. If it was just power cut off, then maybe not. Do you have some reason to suspect overheating? I'd expect a Supermicro server to be well-enough engineered so that absent a few dead fans / clogged heat sinks / running in 100°F data center / etc. it wouldn't overheat. You should be able to watch the temperatures via IPMI. – derobert Jul 17 '18 at 13:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.