Is it possible to deny malloc instead of killing process selected by OOM killer? I think out of memory can be at least logged by the process which is better than being killed. Also, part of the problem is that system is unresponsive for long time (couple of minutes) before killing the offensive process.

Are control groups better tool to guarantee other processes memory?

  • You should read about memory overcommit if you aren't familiar with that.
    – Mat
    Commented Sep 22, 2016 at 12:27

1 Answer 1


You can set limits before starting the process. For example set the virtual (soft) memory limit for the current shell to 1G

ulimit -S 1048576

You can also set such limits system wide, for example I have this:

  • /etc/security/limits.conf:

    *               soft    as             25165824
    *               hard    as             25165824
  • /etc/systemd/system.conf:


To limit all processes to 24G on my 32G systems. Although two processes together could still cause such OOM situation, it never happened in practice.

Regarding "unresponsive for long time before killing the offensive process":

  • Don't use a swap partition, then things will be killed quickly.

Note, actually the malloc (virtual memory) is not the real problem. OOM happens only when the process starts using that malloc'ed memory. Unfortunately on Linux you cannot limit the resident memory. See "memory overcommit":

  • Your suggestion probably wins in the "simplicity" category (vs control-groups for example), but I'm hoping for more dynamic approach like don't overcommit any new pages above 80% of RAM under same assumption that already overcommited ones will likely not be requested. Note that I do not have swap. Page scanning, nasty malloc loops somewhere? Not sure how to debug this when system is unresponsive.
    – sevo
    Commented Sep 23, 2016 at 15:25
  • Maybe it's "unresponsive" is because there is no RAM left for HD cache.
    – rudimeier
    Commented Sep 26, 2016 at 18:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .