How to use a variable with SLURM sbatch to set the output/error file name?

Question

We've just switched to using SLURM and I would like to submit a series of jobs using a loop and sbatch. Previously, I could use a variable as part of the output file names. I've been trying to do this in sbatch using --export to pass in the variable but can't get the variable to be interpolated for the std error/output file names. I think it's working for the job name (-J) and --wrap parts though.

for i in *fastq.gz; do sbatch \
--export=i=$i --error='$i.eo%j' --output='$i.eo%j'  \
-J trim_`basename ${i}` \
--wrap="fastq_trim_single.sh ${i}" \
; done

Produces output files with $i in the name: $i.eo68065. Is it possible to do what I want to do?

Note: ideally, I would like to use basename $i rather than $i for the output file name.

Answer 1

Not sure if this fits your use-case, but have you considered using a job array? When using a job array, the variable SLURM_ARRAY_TASK_ID will be assigned to the current index of the job array, which you could use in the --error and --output commands.

Here are some examples that can be useful: https://slurm.schedmd.com/job_array.html