You can set limits before starting the process. For example set the virtual (soft) memory limit for the current shell to 1G
ulimit -S 1048576
You can also set such limits system wide, for example I have this:
* soft as 25165824
* hard as 25165824
To limit all processes to 24G on my 32G systems. Although two processes together could still cause such OOM situation, it never happened in practice.
Regarding "unresponsive for long time before killing the offensive process":
- Don't use a swap partition, then things will be killed quickly.
Note, actually the malloc (virtual memory) is not the real problem. OOM happens only when the process starts using that malloc'ed memory. Unfortunately on Linux you cannot limit the resident memory. See "memory overcommit":