1

I'm using GNU Parallel to automatically start a large number of jobs and distribute them on the cores of a machine. One job per core.

parallel python3 program.py ::: inputs1*

When one set of jobs is about to finish, the remaining jobs only use a subset of the available cores. It would be nice to start the next set of inputs with parallel so that they only use the unused cores. So at first a subset of all cores but later, once the previous jobs all finished, all available cores (so I can't use --jobs).

If I just start a second command of parallel, it starts jobs also for the cores that are already used by the other command. Is there a nice and easy way to avoid that?

0

Try:

parallel --load 100% ...

It looks at the number of processes currently running and starts a job if there is a core that is idle.

  • I doesn't work. On a machine with 12 cores and 5 jobs running on 5 of them from a previous parallel command, parallel --load 100% ... started 12 new jobs such that more jobs were running than cores were available. – CGFoX Jan 30 '17 at 9:10
  • But where the 5 actually using the CPU? If they were idle, GNU Parallel will ignore them. – Ole Tange Jan 30 '17 at 14:12
  • The 5 cores seemed to be fully used on htop. Or is there another way to check? – CGFoX Jan 30 '17 at 15:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.