My server runs several dozens of processes that are not in my control (mostly monitoring software).
According to my application logs, sometimes some of the third party processes try to allocate too much memory eating up all RAM and swap remaining in the system. My Java application crashes with OOM in this case. I can reproduce this situation with a simple memory consuming script.
Since I cannot control other software running on the machine I decided to limit the memory available to the third party processes using cgroups. That works very well.
My problem is to identify and separate OS/kernel/critical system processes from the third party processes.
Right now, I'm trying to find third party processes and put them into a cgroup with limited memory using some common path patterns in the process command line.
I'm wondering would it be OK to let my application run in the root cgroup (unlimited memory) and place ALL other processes including ALL OS and kernel processes to a cgroup with RAM and SWAP limited to maximum 1GB? (effectively no swap)
I hope that 1GB should be enough for running a RHEL OS (kernel 2.6.32-279.el6.x86_64) and limiting the available RAM to all system processes would not affect the system stability.