3

I am a beginner in the use of .sh scripts so please excuse my ignorance. This is my problem:

To submit my jobs to our cluster the corresponding submit file has to contain a "slurm header" and looks something like this.

#!/bin/sh 
#
########## Begin Slurm header ##########
#
#SBATCH --job-name=blabla
#
########### End Slurm header ##########

# Load module
module load math/matlab/R2020a

# Start a Matlab program
matlab -nodesktop -r "program $1 $2"

exit

Notice that I am passing two arguments to this .sh file which are then passed on to the matlab program. How can I make the job-name in the Slurm header dynamic according to the input arguments?

Simply writing #SBATCH --job-name=blabla$1$2 predictably doesn't work.

11
  • is there some reason to require the job name to be set within the in-file header, and not to e.g. just submit the jobs with something like sbatch --job-name "foo bar" script.sh foo bar?
    – ilkkachu
    Sep 17, 2021 at 14:15
  • @ilkkachu I thought about that a while ago as well. Since my wrapper.sh does a loop like for ii, jj ... do sbatch script.sh $ii $jj I would have to pass the two arguments to script.sh anyways and just making the jobname in the slurm header dynamic seems to keep the wrapper.sh and the script.sh much better human readable (I am very unlikely to do much .sh scripting in the future so coming back later and knowing what I did is a huge plus for me). Or is there a way to load a bunch of slurm options and run both matlab lines in a one liner inside a loop like you suggest, but well readable?
    – stollenm
    Sep 17, 2021 at 15:04
  • Does this answer your question? SLURM: Custom standard output name
    – cas
    Sep 18, 2021 at 4:55
  • You don't have to provide the job name in #SBATCH comments (which are shell comments that are interpreted by the sbatch command, not by bash - you can't use bash variables in them). You can use sbatch's -J or --job-name option (e.g. sbatch -J "blabla$1$2" ...). The #SBATCH comments are just a convenience feature, a way of embedding sbatch command-line options into the job script.
    – cas
    Sep 18, 2021 at 5:01
  • @cas Thanks for the suggested post. However I do need the input variable $1$2 in the job name, not just any dynamic job name.
    – stollenm
    Sep 21, 2021 at 9:48

2 Answers 2

5

Write an sbatch job script like the following, with just the commands you want run in the job:

#!/bin/sh 

# you can include #SBATCH comments here if you like, but any that are
# specified on the command line or in SBATCH_* environment variables
# will override whatever is defined in the comments.  You **can't**
# use positional parameters like $1 or $2 in a comment - they won't
# do anything.

# Load module
module load math/matlab/R2020a

# Start a Matlab program
# give it five arguments, -nodesktop, -r, program, and two
# more that you pass in as arguments to THIS script.
matlab -nodesktop -r "program" "$1" "$2"

# alternatively (since I don't know how matlab runs "program",
# or how it handles args or how it passes them on to a matlab
# script), maybe just three args:
# matlab -nodesktop -r "program $1 $2"

exit

save it as whatever you like - e.g. ./mymatlabjob.sh - and make it executable with chmod +x mymatlabjob.sh

Then run it like so on the command line:

sbatch --job-name "whatever job name you want" ./mymatlabjob.sh arg1 arg2

where arg1 and arg2 are the arguments you want to pass on to the matlab job.

Or in a nested loop like this:

#!/bin/sh

for i in 1 2 3; do
  for j in 3 2 1; do
    sbatch --job-name "blablah$i$j" ./mymatlabjob.sh "$i" "$j"
  done
done

Running that will run 9 different jobs with sbatch, each with different job names - one for each iteration of $i and $j.

8
  • Perfect! This is even better, thanks. Is the "make it executable"-step absolutely needed? If yes, what does it do?
    – stollenm
    Sep 21, 2021 at 10:46
  • reload the page to see the edit, i forgot to pass on the arguments in the first version.
    – cas
    Sep 21, 2021 at 10:51
  • I'm not sure if making it executable is required or not. I can't remember how sbatch and slurm handle job scripts that aren't executable (it may run them anyway, or maybe not). It certainly doesn't hurt to make it executable. chmod +x sets the x execute bit(s) in the file's permissions - this means you can just run it directly without having to run the interpreter (sh or bash) with the script as the first argument. ./myscript.sh (or just myscript.sh if it's in a directory listed in $PATH) rather than bash myscript.sh. The #! line tells linux which interpreter to use.
    – cas
    Sep 21, 2021 at 10:56
  • Thanks again very much. Your help is much appreciated and this is definitely the best way to do it for me. Since the question title and terdons answer match so well however I will leave the accept with terdon so that if people search for the "comment editing sh. scirpt" they find a matching answer.
    – stollenm
    Sep 21, 2021 at 10:57
  • That's fine - he was first to answer with something that works well enough for you. I just wanted to make sure that you understood how sbatch and job names and sh or bash (and command-line options in general) work - because it's much easier to use tools if you understand the basics.
    – cas
    Sep 21, 2021 at 11:01
3

I don't think you can. All lines starting with # are ignored by the shell, and the $1 and $2 are shell things. Many job managers, including slurm, have some commands that are written as shell comments, so ignored by the shell, but are read by the job manager. This is what your SBATCH line is:

#SBATCH --job-name=blabla

So there is no way of doing this dynamically within the same script. However, you can make a wrapper script that does this. For example:

#!/bin/sh
cat <<EoF
#!/bin/sh 
#
########## Begin Slurm header ##########
#
#SBATCH --job-name=blabla$1$2
#
########### End Slurm header ##########

# Load module
module load math/matlab/R2020a

# Start a Matlab program
matlab -nodesktop -r "program $1 $2"

exit
EoF

If you now run this script with two parameters, it will print out the script you actually want:

$ foo.sh param1 param2
#!/bin/sh 
#
########## Begin Slurm header ##########
#
#SBATCH --job-name=blablaparam1param2
#
########### End Slurm header ##########

# Load module
module load math/matlab/R2020a

# Start a Matlab program
matlab -nodesktop -r "program param1 param2"

exit

So you can just do:

foo.sh param1 param2 > slurm_script.sh
3
  • Just out of fun, I tried $(echo "#SBATCH --job-name=$1$2") - but bash then actually tries to execute a command #SBATCH instead of treating the result as comment. Somewhat expected but still -no luck there ;)
    – AdminBee
    Sep 17, 2021 at 10:37
  • @ilkkachu no, not at all. Slurm will only be given the slurm_script.sh script. At the moment, the OP is writing that manually, but this would allow them to do it automatically and change the slurm header as desired.
    – terdon
    Sep 17, 2021 at 11:01
  • @ilkkachu no, I mean that the original process was "manually write a slurm script that takes two arguments" and the problem is that that would not allow the slurm header to be set automatically since that is a comment to the shell. My solution is "automate the writing of the slurm script so that you can set the comment as well". The difference is that the OP will run scriptA to generate scriptB instead of writing scriptB manually.
    – terdon
    Sep 17, 2021 at 11:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .