11

In our cluster, we are restricting our processes resources, e.g. memory (memory.limit_in_bytes).

I think, in the end, this is also handled via the OOM killer in the Linux kernel (looks like it by reading the source code).

Is there any way to get a signal before my process is being killed? (Just like the -notify option for SGE's qsub, which will send SIGUSR1 before the process is killed.)

I read about /dev/mem_notify here but I don't have it - is there something else nowadays? I also read this which seems somewhat relevant.

I want to be able to at least dump a small stack trace and maybe some other useful debug info - but maybe I can even recover by freeing some memory.

One workaround I'm currently using is this small script which frequently checks if I'm close (95%) to the limit and if so, it sends the process a SIGUSR1. In Bash, I'm starting this script in background (cgroup-mem-limit-watcher.py &) so that it watches for other procs in the same cgroup and it quits automatically when the parent Bash process dies.

  • I couldn't find any authority sources, nor I could find a way to invoke OOM killer for specific process manually (to test the idea), but from what I found it seems that OOM killer is simply sends SIGTERM, so you have to set a handler for this signal. – Hi-Angel Dec 6 '15 at 4:06
  • 5
    @Hi-Angel: From the Linux source code, it seems that it sends SIGKILL. – Albert Dec 7 '15 at 11:09
  • @Albert After reading the source code, i also think that OOM Killer will direct send a SIGKILL signal. – andy Dec 24 '15 at 2:31
4

It's possible to register for a notification for when a cgroup's memory usage goes above a threshold. In principle, setting the threshold at a suitable point below the actual limit would let you send a signal or take other action.

See:

https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt

5

OOM killer does send a SIGKILL as it would otherwise be counter-productive to let the problematic program the choice of continuing.

This means that there is absolutely no way for a process to know when it is about to get killed by it.

Managing such issues usually imply making corrections to the programs or their configuration. Sometimes, depending on the system's configuration, simply increasing swap space can give the OS more memory management flexibility to avoid such drastic measures.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.