1

I'm running a linux box(ubuntu) and every few months it freezes and I'm left with no choice(SysRq-F and other keyboard shortcuts are not working) but to reboot it losing all the data.

I'm wondering why the kernel is not killing the processe(s) that eats the memory or CPU?

What's the point keeping the system in a frozen/unresponsive state?

  • 1
    Make sure you have enough swap space. Also it would be useful to have some health status information - to know which parameter is consumed mostly. – German Lashevich Jan 16 '18 at 22:39
  • could you please provide the logs for the period of time you have the issue ? /var/log/messages /var/log/syslog /var/log/kern.log they might provide you interesting information on the cause of the problem. Linux does have a process killing mechanism when things goes to far. – Kiwy Jan 17 '18 at 11:19
3

If the SysRq key combinations don't work when the system is "frozen" and you have verified that they do work when the system is running normally, then the problem is something more serious than resource exhaustion. Might be a kernel bug; might be an intermittent hardware problem. More information is needed.

If the system is server-grade hardware, there might be a separate management processor which can be used to remotely access the system console, get alerts of hardware problems or power on/off the system. Even cheaper server systems might have some kind of a hardware error log. You might need ipmitool or possibly vendor-specific tools to access it, but it might tell you something more.

  • Linux Kernel do shut processes that tend to be irresponsive or eating too much memory. I have several experience with debian 6 7 and 8 that shows precisbly that, but if nothing at all is responsive, it might be a kernel panic or something like that – Kiwy Jan 17 '18 at 11:19
  • I managed a farm of server. Seen many unresponsive aerver due to load, RAM exaustion... even if the OOM kicks in, the chance things go back to normal are usually slim – Rui F Ribeiro Jan 18 '18 at 8:27
  • I've seen those too, but if it's just load or RAM exhaustion, the "magic SysRq" keys should still work, as they are 100% kernel functions. If they don't work, the kernel is either no longer running or stuck in a tight loop. Then you'll need something like hardware error logs or kdump + a hardware NMI button to get more information. – telcoM Jan 18 '18 at 10:20
  • @telcom I am more worried about things at large scale, cannot babysitting non-critical servers where I know I will have extraordinary regular occurrences of these situations. I just have a watching doing reboots-for instance I have a student server that when students have to deliver homework might reboot alone maybe once every 2 weeks. – Rui F Ribeiro Feb 12 '18 at 11:23
2

It would be interesting to learn what the server does and what resources does it have; some configurations might be tuned or extra resources might be needed to alleviate the problem of the server being in an unresponsive state; or a process monitored and restarted often in a while; your post is not clear and that would be the first approach.

However, as a countermeasure for recovering an unresponsive server there is what is known as a watchdog.

If the machine is physical you can even buy an hardware watchdog supported by the linux kernel.

Some iOT devices even come with a watchdog already built-in in the SoC.

For either a physical server or a VM you are able to, and I have in the past, configured software watchdogs for problematic servers (for instance, a student server, a machine doing calculations and even dodge coin mining (not myself!) ) - because I preferred a reboot than having those services unresponsive.

The software watchdog is basically a periodic interrupt monitoring some resources of the server and an application talking with it. When the configured verifications fail, and/or the application stops sending data, it forces a reboot of the server.

See The Linux Kernel/Softdog Driver

What is a Watchdog Timer?

A watchdog timer is a device that triggers a system reset if it detects that the system has hung. A program running on the system is supposed periodically to service the watchdog timer by writing a "service pulse." If the watchdog is not serviced within a particular period of time, the watchdog assumes that the system has hung, and triggers a system reset.

What is Softdog?

Usually, watchdog timers are implemented as add-on cards, or as on-chip peripherals within microcontrollers. But if there is no hardware watchdog, the Linux kernel can provide a software watchdog implemented using kernel timers.

Linux Watchdog Mechanism

In Linux, the watchdog driver provides a character driver interface to the user space. When some data is written to the watchdog driver, the watchdog driver services the watchdog hardware. The user space application periodically writes some data to the watchdog driver, depending upon the watchdog timeout period. If for some reason the user space application hangs, the watchdog device does not get serviced and hence triggers a system reset.

Usually the application that writes to the watchdog driver is a watchdog daemon which monitors processes in the system, as well as other parameters such as CPU utilization, memory utilization, and so on.

How Softdog Works

When the softdog driver is opened, softdog schedules a kernel timer to expire after a specified timer margin. When some data is written to the driver, the softdog driver re-schedules the timer. The user space watchdog daemon periodically writes to the driver, and the timer is continuously rescheduled and hence the timer callback is never called. If the watchdog daemon stops writing to the driver, the timer expires and the callback is called. In the timer callback, the system is restarted.

see also Debian - Package: watchdog - system health checker and software/hardware watchdog handler

The watchdog program writes to /dev/watchdog every ten seconds. If the device is opened but not written to within a minute, the machine will reboot. This feature is available when the kernel is built with "software watchdog" support (standard in Debian kernels) or if the machine is equipped with a hardware watchdog (in which case this package can also be used to "pet" it, resetting its timer).

The kernel software watchdog's ability to reboot will depend on the state of the machine and interrupts.

The watchdog tool itself runs several health checks and acts appropriately if the system is not in good shape.

I would also advise configuring the syslog daemon for sending logs to a remote syslog server to have logs to peruse whenever the system is working properly or not.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.