In our cluster, we are restricting our processes resources, e.g. memory (
I think, in the end, this is also handled via the OOM killer in the Linux kernel (looks like it by reading the source code).
Is there any way to get a signal before my process is being killed? (Just like the
-notify option for SGE's
qsub, which will send
SIGUSR1 before the process is killed.)
I want to be able to at least dump a small stack trace and maybe some other useful debug info - but maybe I can even recover by freeing some memory.
One workaround I'm currently using is this small script which frequently checks if I'm close (95%) to the limit and if so, it sends the process a
SIGUSR1. In Bash, I'm starting this script in background (
cgroup-mem-limit-watcher.py &) so that it watches for other procs in the same cgroup and it quits automatically when the parent Bash process dies.