I'm running some training using GNU parallel. The training takes about 30 secs to run one iteration, and I need to run about 3000. The training itself can't be parallelized (at least not without considerable effort), however, I run this training with several different seeds, and all these seeds can easily run on different cores.

This is how I am using parallel -

parallel ./train.py config/config.yml _results/ \
--seed {1} \
::: {1..5}

When I run this, parallel puts all 5 processes on the same core (core0), and each of them has 20% CPU usage (as checked using htop).

If I run another set of training using the same command, 5 more processes get added to core0, and now they all show 10% CPU usage.

I am on Ubuntu 18.04

Operating System: Ubuntu 18.04.4 LTS
          Kernel: Linux 5.3.0-28-generic

and Ryzen 5 3600

processor   : 0
vendor_id   : AuthenticAMD
cpu family  : 23
model       : 113
model name  : AMD Ryzen 5 3600 6-Core Processor
stepping    : 0
microcode   : 0x8701013
cpu MHz     : 3868.329
cache size  : 512 KB
physical id : 0
siblings    : 12
core id     : 0
cpu cores   : 6
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 16
wp      : yes

My current (non-)solution is to use taskset after starting the training to put each seed on a different core.


GNU Parallel does not control which core to run on, so I will expect these will also run on the same core:

bzip2 < /dev/zero >/dev/null &
bzip2 < /dev/zero >/dev/null &
bzip2 < /dev/zero >/dev/null &
bzip2 < /dev/zero >/dev/null &
bzip2 < /dev/zero >/dev/null &

If that is the case, something is forcing your shell to spawn all processes on the same core and you should probably find out what causes this.

But if taskset works, you can do this as a workaround:

parallel taskset -c '{=$_=slot()-1=}' train...
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.