I have a single machine (Ubuntu 16.04 Server) with 4 TitanX GPUs. This will be a lab machine on which students will learn about CUDA and stuff. I installed SLURM because I want a tool to schedule and enqueue jobs automatically based on GPU availability.

For this to work, students should have access to GPUs only through SLURM. Is there a way to achieve this in single-machine setup, where SLURM and cards are on the same host?

Students should only access cards through:

# srun --gres ...

I know about CUDA_VISIBLE_DEVICES but this variable is ignores when set by root or even each user. It only works when set per process, for example deviceQuery from CUDA Samples:

# CUDA_VISIBLE_DEVICES=2 /usr/local/cuda-9.0/extras/demo_suite/deviceQuery

And it might be used by the SLURM itself to restrict cards, so it is (rather) not a solution.

Best Regards

  • 1
    I'm pretty sure all access to the nVidia cards goes through /dev/nvidia*, so it ought to be doable with permissions. Having never used any of the software involved, though, I'll just leave this as a pointer, I won't hazard to guess how to implement. – derobert Apr 13 '18 at 18:05
  • Without completely disabling logins to the node that contains the GPUs, I don't know of an easy way to do with within Slurm – Tablemaker Aug 15 at 15:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.