I have a single machine (Ubuntu 16.04 Server) with 4 TitanX GPUs. This will be a lab machine on which students will learn about CUDA and stuff. I installed SLURM because I want a tool to schedule and enqueue jobs automatically based on GPU availability.

For this to work, students should have access to GPUs only through SLURM. Is there a way to achieve this in single-machine setup, where SLURM and cards are on the same host?

Students should only access cards through:

# srun --gres ...

I know about CUDA_VISIBLE_DEVICES but this variable is ignores when set by root or even each user. It only works when set per process, for example deviceQuery from CUDA Samples:

# CUDA_VISIBLE_DEVICES=2 /usr/local/cuda-9.0/extras/demo_suite/deviceQuery

And it might be used by the SLURM itself to restrict cards, so it is (rather) not a solution.

Best Regards

  • 1
    I'm pretty sure all access to the nVidia cards goes through /dev/nvidia*, so it ought to be doable with permissions. Having never used any of the software involved, though, I'll just leave this as a pointer, I won't hazard to guess how to implement.
    – derobert
    Commented Apr 13, 2018 at 18:05
  • Without completely disabling logins to the node that contains the GPUs, I don't know of an easy way to do with within Slurm
    – Tablemaker
    Commented Aug 15, 2019 at 15:50

1 Answer 1


create a PrologSlurmctld (https://slurm.schedmd.com/slurm.conf.html#OPT_PrologSlurmctld) script which set the access to the GPU (/dev/nvidia* devices) with setfacl to the current user

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .