1

So, I am by no means a sysadmin but I need to use an existing SLURM installation to launch a sizable amount of jobs (around 5000).

The cluster is composed of 1 node with 10 GPUs (with 8GB of memory each) and 56 CPUs.

Every job is a batch script that I run with sbatch <file> and then I use sview to see what's going on

These jobs need to run on a GPU but they have rather low GPU memory requirements (around 2GB) so I figured I could fit 3 of them on a single GPU.

I have been pulling my hair trying to find a way to allow 3 jobs to run on the same GPU at the same time but the documentation is so massive that I feel completely lost.

I am not sure if I am even using this properly.

Jobs look like this:

#!/bin/bash
#SBATCH --time=00:10:00                 
#SBATCH -p n1                        
#SBATCH --nodes=1                       
#SBATCH --ntasks=1                      
#SBATCH --cpus-per-task=[nb_cpu]               
#SBATCH --mem-per-gpu=1                        
#SBATCH --job-name=[job_name]           
#SBATCH --mail-user=[list_mail]  
#SBATCH --mail-type=NONE                 
#SBATCH --gid=dl
#SBATCH --output=[folder]/%x.log    

echo "Current GPU: $CUDA_VISIBLE_DEVICES"
# a bunch of initialization
module load opencv/4.5.4 deeplearning
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/raid1/CLUSTER/LIB/custom_libs/
export OPENCV_LOG_LEVEL=DEBUG
export OPENCV_VIDEOIO_DEBUG=1
run_dl --nn-dir=/home/dl/networks --root-dir=[root_dir] [file]
if [ $? -eq 0 ]
then
    mkdir -p [folder]/done
    echo [id] > [folder]/done/[job_name]
else
    mkdir -p [folder]/fail
    echo [id] > [folder]/fail/[job_name]
fi

These jobs are generated by a script and can be modified easily if needed. Am I even going the right direction by using sbatch ? I am quite overwhelmed.

2
  • 2
    AFAIK (but note that I haven't used slurm professionally as a sysadmin for nearly 10 years now), slurm assigns only one job to a GPU at a time. It will run jobs in parallel if you have multiple GPUs that can run the jobs, otherwise it runs them in series as a GPU becomes available. I suspect you'll have to write your actual code to do three or so different "jobs" at once, then slurm can schedule it as one job on a GPU....then divide up your total jobs by three so you can run 5000/3 slurm jobs.
    – cas
    Nov 26, 2022 at 10:46
  • There is GPU "sharding" (since Slurm 22.05). See my answers for details.
    – Jongwook Choi
    Apr 10 at 12:10

1 Answer 1

Reset to default
2

You should use "Sharding" GRES (gres:shard) instead of gres:GPU, available in 22.05 or newer.

https://slurm.schedmd.com/gres.html#Sharding

It allows different jobs to share a GPU -- just like oversubscribed Cores and RAM resources. The conventional gres:gpu exclusively allocates a GPU to jobs no matter how much memory is used.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .