I have a single machine (Ubuntu 16.04 Server) with 4 TitanX GPUs. This will be a lab machine on which students will learn about CUDA and stuff. I installed SLURM because I want a tool to schedule and enqueue jobs automatically based on GPU availability.
For this to work, students should have access to GPUs only through SLURM. Is there a way to achieve this in single-machine setup, where SLURM and cards are on the same host?
Students should only access cards through:
# srun --gres ...
I know about
CUDA_VISIBLE_DEVICES but this variable is ignores when set
by root or even each user. It only works when set per process, for example
deviceQuery from CUDA Samples:
# CUDA_VISIBLE_DEVICES=2 /usr/local/cuda-9.0/extras/demo_suite/deviceQuery
And it might be used by the SLURM itself to restrict cards, so it is (rather) not a solution.