0

Today I was working on a remote machine (the likes of which I'll likely never see), and it was hung, so I told the user to restart it. The machine came up again, and I ssh'd into it, but as soon as I did I was presented with a message about it's temperature being above 60°C, and so the kernel shut the machine down again, and of course I was disconnected from ssh.

We're about to boot the machine back up again, is there anyway to read things from a log that would state that in the past this machine had shut down because it was too hot?

  • What version of what distribution? Your one word answer is "Probably", but the location to look may be different between distributions, especially if the install is old. Most people here would prefer to give an answer that was right from the start, without writing a very long document explaining all of the possibilities. – Ed Grimm Feb 22 '19 at 8:04
  • Do you use systemd? – jimmij Feb 22 '19 at 18:30
  • @jimmij yes, I do. – leeand00 Feb 23 '19 at 1:06
  • So did you searched journalctl? – jimmij Feb 23 '19 at 8:22
  • @jimmij No I guess not...what should I grep for? Something about temperature? – leeand00 Feb 24 '19 at 17:14
2

When the machine overheats, before kernel shuts down the system, it produces a message stored in the log file. For systemd setup this file is accessible via journalctl command and the relevant output should look like this:

Jul 09 05:28:39 jjmach kernel: thermal thermal_zone0: critical temperature reached (96 C), shutting down
Jul 09 05:28:40 jjmach root[27818]: ACPI event unhandled: thermal_zone LNXTHERM:00 000000f0 00000001
Jul 09 05:28:57 jjmach kernel: thermal thermal_zone0: critical temperature reached (96 C), shutting down
Jul 09 05:28:57 jjmach lightdm[174]: Failed to get D-Bus connection
Jul 09 05:28:46 jjmach pppd[17312]: Hangup (SIGHUP)
Jul 09 05:29:00 jjmach org.a11y.Bus[265]: Reloaded configuration
Jul 09 05:28:42 jjmach systemd[1]: Stopping Daemon for power management...

So we need to search for 'critical temperature' stuff in the $SYSTEMD_PAGER (usually less) or directly:

journalctl -g 'temperature|critical'

You can add -b -1 to search only in messages of the boot before last.


With that being said, I think 60°C is really very low for tripping point. Most CPUs often reach that temperature during compilation or other resourceful tasks. Be sure you have all proper modules loaded (for you processor) and check thermal parameters with sensors-detect followed by sensors command.

  • 1
    For a rack-mount server, sensors for monitoring the temperature of ambient air or input airflow are actually reasonably common, and 60°C is a common tripping point for that: if the ambient air is at that temperature, the server room probably has had a major HVAC failure and system cooling is going to be compromised, so the system shuts down to protect itself. Such servers often have hardware error logs: try ipmitool sel or vendor-specific tools to access the hardware error log. – telcoM Feb 25 '19 at 9:18
  • 1
    Servers might also have a BIOS setting that allows the administrator to decide what is going to happen on an overheat condition. At least on Fujitsu servers, it is possible to set them to not shut down, and instead attempt to keep running until physically destroyed; that might be a valid choice if the server is a part of a cluster that is providing a life-or-death critical service. However, servers might also include a non-resettable telltale that remembers if the server has been allowed to overheat; and if so, server warranty will probably be void. – telcoM Feb 25 '19 at 9:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.