Is it possible to limit the memory usage of all processes started by GNU parallel? I realize there are ways to limit the number of jobs, but in cases where it isn't easy to predict the memory usage ahead of time it can be a difficult to tune this parameter.

In my particular case I'm running programs on a HPC where there are hard limits on process memory. E.g. if there's 72GB of ram available on a node, the batch system will kill jobs that exceed 70GB. I'm also unable to spawn jobs directly to the swap and hold them there.

The GNU parallel package comes with niceload, which seems to allow for the current memory usage to be checked before a process runs. However I'm not sure how to use this.

2 Answers 2


The short answer is:

ulimit -m 1000000
ulimit -v 1000000

which will limit each process to 1 GB RAM.

Limiting the memory the "right" way is in practice extremely complicated: Let us say you have 1 GB RAM. You start a process every 10 seconds and each process uses 1 MB more every second. So after 140 seconds you will have something like this:


This sums up to 1050 MB RAM, so now you need kill something. What is the right job to kill? Is it 140 (assuming it ran amok)? Is it 10 (because it has run the least amount of time)?

In my experience jobs where memory is an issue are typically very predicable (e.g. transforming a bitmap) or very little predictable. For the very predictable ones you can do the computation beforehand and see how many jobs can be run.

For the unpredictable you ideally want the system to start few jobs that take up a lot of memory, and when they are done, you want the system to start more jobs that take up less memory. But you do not know beforehand which jobs will take a lot, which will take a little, and which ones run amok. Some jobs normal life cycle is to run with little memory for a long time and then balloon to a much bigger size later on. It is very hard to tell the difference between those jobs and jobs that run amok.

When someone points me to a well thought out way to do this in a way that will make sense for many applications, then GNU Parallel will probably be extended with that.

  • That is the problem I'm having, each process spawned by parallel takes anywhere from 70 to well over 100 hours. It seems that the memory usage stays at the amount I predicted but it expands sometime later causing my usage on that node to exceed the limit, however because the batch only watches the node usage it isn't possible to tell which particular job caused the problem.
    – Joe
    Dec 12, 2014 at 14:02
  • When you come up with a clever algorithm that "does the right thing" please let me know.
    – Ole Tange
    Dec 13, 2014 at 8:42
  • Maybe a practical approach would be: While less than X GB RAM free: Kill the youngest child and put it back on the queue. Only spawn new jobs when 3*X GB free. It will waste CPU time on the jobs that needs to be rerun. Another approach would be to suspend (not kill) the young jobs. That would make them swap out, and they would need to swap in eventually - which may be even slower. The killing should not be too hard to implement.
    – Ole Tange
    Dec 13, 2014 at 9:08
  • This is more or less what I'd like. The idea would be to start jobs till X% of the memory is used, then halt the spawning till one job completes. I think that niceload is what I'm looking for but I'm at a loss how to implement it with parallel.
    – Joe
    Dec 29, 2014 at 15:46
  • The above cannot be done with niceload. Try --memfree which in the git-version of GNU Parallel. Let me know how it works out for you.
    – Ole Tange
    Dec 29, 2014 at 16:23

Things have changed since 2014.

Git version e81a0eba now has --memsuspend

--memsuspend size (alpha testing)

Suspend jobs when there is less than 2 * size memory free. The size can be
postfixed with K, M, G, T, P, k, m, g, t, or p which would multiply the size
with 1024, 1048576, 1073741824, 1099511627776, 1125899906842624, 1000,
1000000, 1000000000, 1000000000000, or 1000000000000000, respectively.

If the available memory falls below 2 * size, GNU parallel will suspend some
of the running jobs. If the available memory falls below size, only one job
will be running.

If a single job takes up at most size RAM, all jobs will complete without
running out of memory. If you have swap available, you can usually lower
size to around half the size of a single jobs - with the slight risk of
swapping a little.

Jobs will be resumed when more RAM is available - typically when the oldest
job completes.
  • Is there a way to limit memory to some percent of total available? I think this would be better in contexts where job size and memory per job aren't know a priori Dec 3, 2023 at 3:42
  • @BrittonKerin Currently: no. But it would not be impossible to implement. I assume you want this behavior: If the system has 10GB RAM, then --memsuspend 20% should do exactly the same as --memsuspend 2G so GNU Parallel simply does the calculation for you. Is that correctly understood?
    – Ole Tange
    Dec 4, 2023 at 12:53
  • yes I did it for linux using MemTotal from /proc/meminfo but doubt that works on BSD etc. % just seems like a better attemtp to leave a system otherwise operational while parallel runs than flat amount. Btw did you see bug report on email list? --memsuspend=50% is presently tolerated as is garbage after a supported amount e.g. --memsuspend=50kfoo Dec 7, 2023 at 13:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .