Is there a way to run a startup (or teardown) script on each node when using GNU Parallel?

For example, I wish to distribute (and balance) many single-threaded tasks among many multi-core nodes (specifically using PBS to assign which nodes, from a shared HPC cluster). For each new node, it is necessary to run some set-up commands (e.g. loading environment modules) once-off before running tasks.

  • Should I simply accept the inefficiency of making every individual task reload the environment modules? – benjimin Aug 29 '17 at 1:15
  • Is there an alternative to GNU parallel? xargs -P is great on a single machine; is there another simple tool for balancing many trivially-parallisable tasks over multiple hosts? – benjimin Aug 29 '17 at 1:18
  • Possibly --env of env_parallel has some similar effects? – benjimin Aug 29 '17 at 2:15

Can you use --nonall?

parallel --nonall --slf $PBS_NODEFILE setup.sh

This will not work if it is setting up environment. For that you could try:

env_parallel --record-env
[load the environment]
seq 100 | env_parallel --slf $PBS_NODEFILE --env _ echo do the job
  • you mean -S $(cat $PBS_NODEFILE) or --sshlogin $PBS_NODEFILE ? – benjimin Aug 29 '17 at 7:23
  • Just realised cluster does not readily permit ssh between nodes (but changing the ssh command might otherwise be a way to get the effect I asked for). Not sure if possible to create a ssh-like command using pbsdsh. Is there an MPI version of parallel? – benjimin Aug 29 '17 at 7:25
  • Sorry: --slf $PBS_NODEFILE. – Ole Tange Aug 29 '17 at 10:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.