I am trying to distribute independent runs of a process using GNU Parallel on a HPC that uses Slurm workload manager. Briefly, here is the data analysis set up:
./myscript --input infile.txt --setting 1 --output out1 ./myscript --input infile.txt --setting 2 --output out2 ./myscript --input infile.txt --setting 3 --output out3 ./myscript --input infile.txt --setting 4 --output out4
#SBATCH --time=00:02:00 #SBATCH --nodes=2 #SBATCH --cpus-per-task=2 cat myCommands | parallel -j 4
This works, however it only uses one node. The two cores on that nodes are split into 4 threads to make room for 4 jobs as requested by parallel. That is not desirable.
My searching indicates I will need a
nodefile and a
sshloginfile to accomplish this, but I see no examples online that work with
Slurm, only with
How can I make the script (1) use both nodes, and (2) not split cores into threads?