I have a submit script looks like below, it tries to run a large number of instances of csce.py in backgrounds with 3 nodes.... in a laptop, this usually could successfully automatically distribute all the background tasks into 16 cores.... However, I am not sure if in a cluster, it would also automatically distribute the 4*13*9 tasks in 3 nodes (48 cores).

#SBATCH -N 3                   # Total number of nodes requested (16 cores/node)
#SBATCH -n 48                  # Total number of mpi tasks requested

for simplify in  0.1 0.15 0.2 0.25
do for lmbda in 0.5 1 2  5 10 20  50 100 200 500 1000 2000 5000
do for mu in 0.005 0.01 0.05 0.1 0.5 1 5 10 50 
do rm eci.out
csce.py --mu $mu --lmbda $lmbda --simplify $simplify --favor-low-energy 0.01 --bias-stable --save-energies lmbda_$lmbda\_mu_$mu\_simplify_$simplify\_ce-energies.dat --save-weights lmbda_$lmbda\_mu_$mu\_simplify_$simplify\_ce-weights.dat  --casm-eci-file eci.in lmbda_$lmbda\_mu_$mu\_simplify_$simplify\_eci.out --save-hull lmbda_$lmbda\_mu_$mu\_simplify_$simplify\_ce-hull.dat  --preserve-ground-state 10000 2> lmbda_$lmbda\_mu_$mu\_simplify_$simplify\_error 1> lmbda_$lmbda\_mu_$mu\_simplify_$simplify\_output &


No, if you would have multiple nodes (machines) there is nothing in there that takes advantages of that, everything will run on the machine you run this script.

The & at the end of the csce.py line just makes the operation run in the background on the current machine. So wit this setup you will get 4x12x9 tasks running in parallel on your current machine.

GNU parallel supports remote execution, for that you need to setup automated access to the other machines and think about how any input data is accessed (if it is not stored for reading on some volume shared by all machines you might need to copy the data to work on).

  • so, do you mean that this process would only use 1 node instead of 3 nodes? or even worse, only on 1 core instead of 16 cores? – user40780 Mar 31 '16 at 12:34
  • Only one node, with mulitple processes per core ( 27 to start with). That is unless there is something in csce.py that distributes itself. – Anthon Mar 31 '16 at 13:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.