1

I'm working on a computer with 10k cores, but only allowed to access 1,000 at a time.

I have a script that could benefit from GNU parallel in multiple places. Processing level A in parallel, and within that script, do something 30x.

It will take a lot of work to re-write the entire script to use the parallel --link and ::: A B C ::: $(seq 30) syntax.

Is there a way that two independent calls to parallel can communicate enough that they limit the total number of jobs to 1000 between the two of them?

Improve this question

2 Answers 2

Reset to default
2

If I understand correctly, you want to run 2 (or more instances) of GNU Parallel, and you want the total number of running jobs to be <1000.

So at some point one of the may run 300 jobs, then the other should limit to 700 jobs.

--limit is made for this kind of specialized situation: It will run a script of your choice and based on the exit value it will limit the number of jobs.

So now you need some way of determining the total number of jobs running.

Maybe some kind of [ $(ps aux | grep myprogram | wc -l) -gt 1000 ]?

(What kind of monster has 10k cores?)

Improve this answer
3
  • I was wondering that myself. Best I can come up with is an Intel Platinum 8280 or AMD EPYC 7003. 128 cores each. If you can get a 4-socket board that takes you up to 512 cores. Factor of 20 short of what the OP says he has.
    – doneal24
    Jul 27, 2022 at 13:46
  • Actually 5120 cores but 2 threads/core. curc.readthedocs.io/en/latest/clusters/alpine/index.html
    – mankoff
    Jul 27, 2022 at 17:04
  • @mankoff Ahhh, so it is just a cluster of 80 nodes - not a single machine. In that case my solution will likely not work directly. But maybe you can look at the number of ssh connections from the controlling node.
    – Ole Tange
    Jul 27, 2022 at 20:31
1

Another solution might be to use --sqlmaster/--sqlworker.

Spawn 1000 --sqlworkers and put the jobs to be run into the database with --sqlmaster.

master1$ seq 10000 | parallel --sqlmaster +pg://user:pass@server/db/table sleep 1.1{}
# The + is needed to append to existing table
master2$ seq 4000 | parallel --sqlmaster +pg://user:pass@server/db/table sleep 1.2{}

worker1..1000$ parallel --sqlworker +pg://user:pass@server/db/table

I think this is less sensitive than the --limit version - especially because you are running on a cluster - not a single machine.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .