So, I am by no means a sysadmin but I need to use an existing SLURM installation to launch a sizable amount of jobs (around 5000).
The cluster is composed of 1 node with 10 GPUs (with 8GB of memory each) and 56 CPUs.
Every job is a batch script that I run with
sbatch <file> and then I use
sview to see what's going on
These jobs need to run on a GPU but they have rather low GPU memory requirements (around 2GB) so I figured I could fit 3 of them on a single GPU.
I have been pulling my hair trying to find a way to allow 3 jobs to run on the same GPU at the same time but the documentation is so massive that I feel completely lost.
I am not sure if I am even using this properly.
Jobs look like this:
#!/bin/bash #SBATCH --time=00:10:00 #SBATCH -p n1 #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --cpus-per-task=[nb_cpu] #SBATCH --mem-per-gpu=1 #SBATCH --job-name=[job_name] #SBATCH --mail-user=[list_mail] #SBATCH --mail-type=NONE #SBATCH --gid=dl #SBATCH --output=[folder]/%x.log echo "Current GPU: $CUDA_VISIBLE_DEVICES" # a bunch of initialization module load opencv/4.5.4 deeplearning export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/raid1/CLUSTER/LIB/custom_libs/ export OPENCV_LOG_LEVEL=DEBUG export OPENCV_VIDEOIO_DEBUG=1 run_dl --nn-dir=/home/dl/networks --root-dir=[root_dir] [file] if [ $? -eq 0 ] then mkdir -p [folder]/done echo [id] > [folder]/done/[job_name] else mkdir -p [folder]/fail echo [id] > [folder]/fail/[job_name] fi
These jobs are generated by a script and can be modified easily if needed. Am I even going the right direction by using
sbatch ? I am quite overwhelmed.