5

I need to run some memory heavy tests in a remote computer through SSH. Last time I did this, the computer stopped responding, and it was necessary for someone to physically reboot it.

Is there a way I can set it up so that the system restarts instead of freezing if too much memory is being used? (I do have root access). The kernel version is 4.9.0.

5
  • Schedule a reboot with at, run your tests, cancel the at command if you're still connected; or wait for the reboot if you aren't. Old sysadmin trick. May 24, 2017 at 15:13
  • 1
    Or see if there is some sort of watchdog support in the BIOS in the event Linux is really failing about and does need to be poked.
    – thrig
    May 24, 2017 at 15:17
  • Do you have any /dev/watchdog* devices?
    – derobert
    May 24, 2017 at 15:41
  • 1
    @SatoKatsura if the system really has become unresponsive who's to say your at job could even be started? May 24, 2017 at 21:15
  • @roaima If the system (presumed to be Linux) has become unresponsive as a result of running out of memory, then OOM is already "committing random acts of terrorism", as DJB used to put it. Then at is just as likely to be left alive to do its job as any other recovery method. There's always a chance of getting stuck. In order to avoid OOM going on a rampage one needs either a hardware watchdog (which is rarely found on consumer motherboards), or setting sane limits. But setting limits doesn't seems to be what the OP wants. May 25, 2017 at 2:53

2 Answers 2

5

To monitor/recover the control of a "unstable"/starver server, I would advise to use an hardware, or failing that a software watchdog; in Debian you can install it with:

sudo apt-get install watchdog

Then you edit /etc/watchdog.conf and add thresholds or tests; from the top of my head, the watchdog is also activated as such that if the kernel does not see it for a good while it reboots. e.g. if a software routine does not talk in a fixed time with /dev/watchdog0 or something similar.

For instance, you can define load thresholds in /etc/watchdog.conf:

max-load-1             = 40
max-load-5             = 18
max-load-15            = 12

Be aware also that some boards/chipsets come with built-in watchdogs; if I am not wrong the Arm A20 is one of them.

From man watchdog

The Linux kernel can reset the system if serious problems are detected. This can be implemented via special watchdog hardware, or via a slightly less reliable software-only watchdog inside the kernel. Either way, there needs to be a daemon that tells the kernel the system is working fine. If the daemon stops doing that, the system is reset.

watchdog is such a daemon. It opens /dev/watchdog, and keeps writing to it often enough to keep the kernel from resetting, at least once per minute. Each write delays the reboot time another minute. After a minute of inactivity the watchdog hardware will cause the reset. In the case of the software watchdog the ability to reboot will depend on the state of the machines and interrupts.

The watchdog daemon can be stopped without causing a reboot if the device /dev/watchdog is closed correctly, unless your kernel is compiled with the CONFIG_WATCHDOG_NOWAYOUT option enabled.

see also Raspberry Pi and Arduino: Building Reliable Systems With WatchDog Timers

3
  • 1
    I tried this but still got a frozen computer that didn't reboot. Jul 27, 2017 at 5:09
  • 1
    @user7000 It is a stress situation and many things can go wrong. It wont work 100% of the time. Also there are hardware watchdogs you can buy. Jul 27, 2017 at 6:44
  • Yeah I've not seen good reviews for any hardware watchdog but I would like to get one. Jul 29, 2017 at 3:39
2

To answer your specific question, you can set sysctls like:

vm.panic_on_oom=1
kernel.panic=10

so that the kernel will panic if the system runs out of memory and then reboot after ten seconds of panicking.

On new systems with full cgroups2 support, systemd-oomd may be a less drastic option.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .