Limit total number of GNU-parallel jobs between mutliple calls

Question

I'm working on a computer with 10k cores, but only allowed to access 1,000 at a time.

I have a script that could benefit from GNU parallel in multiple places. Processing level A in parallel, and within that script, do something 30x.

It will take a lot of work to re-write the entire script to use the parallel --link and ::: A B C ::: $(seq 30) syntax.

Is there a way that two independent calls to parallel can communicate enough that they limit the total number of jobs to 1000 between the two of them?

Answer 1

If I understand correctly, you want to run 2 (or more instances) of GNU Parallel, and you want the total number of running jobs to be <1000.

So at some point one of the may run 300 jobs, then the other should limit to 700 jobs.

--limit is made for this kind of specialized situation: It will run a script of your choice and based on the exit value it will limit the number of jobs.

So now you need some way of determining the total number of jobs running.

Maybe some kind of [ $(ps aux | grep myprogram | wc -l) -gt 1000 ]?

(What kind of monster has 10k cores?)

Answer 2

Another solution might be to use --sqlmaster/--sqlworker.

Spawn 1000 --sqlworkers and put the jobs to be run into the database with --sqlmaster.

master1$ seq 10000 | parallel --sqlmaster +pg://user:pass@server/db/table sleep 1.1{}
# The + is needed to append to existing table
master2$ seq 4000 | parallel --sqlmaster +pg://user:pass@server/db/table sleep 1.2{}

worker1..1000$ parallel --sqlworker +pg://user:pass@server/db/table 

I think this is less sensitive than the --limit version - especially because you are running on a cluster - not a single machine.