Prevent the machine being slowed down by running out of memory

Question

My software runs a command that looks something like:

find | xargs do a potentially memory hungry job

The problem is that sometimes a potentially memory hungry job gets too hungry, the system gets unresponsive and I have to reboot it. My understanding is that it happens due to the memory allocation over commitment. What I would like to happen is that if a job spawned by xargs wants more memory than is available, it dies (I am OK with it) and that is it. I guess I can get this behavior if I turn off overcommitment system-wide, but it is not an option. Is it possible to turn it off for a process?

A possible solution I was thinking of was to set

ulimit -v RAM size

But something tells me it is not a good idea.

Answer 1

I think what you are looking for is --memfree in GNU Parallel:

find ... | parallel --memfree 1G dostuff

This will only start dostuff if there is 1G RAM free. It will start one more until there is either less than 1G RAM free or 1 job running per CPU thread. If there is 0.5G RAM free (50% of 1G RAM) the youngest job will be killed. So in metacode:

limit = 1G
while true:
  if freemem > limit:
    if count(running_jobs) < cpu.threads():
      another_job.start()
  if freemem < 0.5 * limit
    youngest_job.kill()

If combined with --retries 10 you can tell GNU Parallel to retry the killed job 10 times.

If dostuff takes a while to gobble up the memory, use --delay 30s to wait 30s before spawning the next job.

Answer 2

sysctl vm.overcommit_memory 2

If you want to avoid cgroups