1

I recently switched to slurm and looking for a job submission tool, that behaves similar to qsub:

  1. It takes input through a pipe
  2. It prints the output to stdout

Example:

for n in `seq 1 10`; do 
    echo "echo $n" | qsub
done

should send each echo command to a cluster and the output should be 1..10 presumably in random order.

So far I can

  1. send jobs with sbatch in parallel, but not sure to get the output to stdout
  2. send jobs with srun, but then it operates sequentially one by one

Any suggestions?

1 Answer 1

0

Untested:

seq 10 | parallel -j0 -q srun 'echo begin {};sleep 0.{};echo done {}'

If that works, use --keep-order to keep output in correct order:

seq 10 | parallel -j0 -q --keep-order srun 'echo begin {};sleep 0.{};echo done {}'
1
  • Thanks, appreciate the suggestion, but I believe it's doing not what I want: it will spin 10 processes on a local machine to start srun's. Say, I want 10K jobs - running all those locally feels very wrong, even though they just call srun.
    – LazyCat
    Mar 8 at 23:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .