1

I'm using GNU Parallel to automatically start a large number of jobs and distribute them on the cores of a machine. One job per core.

parallel python3 program.py ::: inputs1*

When one set of jobs is about to finish, the remaining jobs only use a subset of the available cores. It would be nice to start the next set of inputs with parallel so that they only use the unused cores. So at first a subset of all cores but later, once the previous jobs all finished, all available cores (so I can't use --jobs).

If I just start a second command of parallel, it starts jobs also for the cores that are already used by the other command. Is there a nice and easy way to avoid that?

1 Answer 1

0

Try:

parallel --load 100% ...

It looks at the number of processes currently running and starts a job if there is a core that is idle.

3
  • I doesn't work. On a machine with 12 cores and 5 jobs running on 5 of them from a previous parallel command, parallel --load 100% ... started 12 new jobs such that more jobs were running than cores were available. Jan 30, 2017 at 9:10
  • But where the 5 actually using the CPU? If they were idle, GNU Parallel will ignore them.
    – Ole Tange
    Jan 30, 2017 at 14:12
  • The 5 cores seemed to be fully used on htop. Or is there another way to check? Jan 30, 2017 at 15:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .