I have a server with 4 GPUs/128 cores where I use a workload manager (slurm) to manage resource allocation between users for computation.
The issue is that the scheduler server (for job submission) runs on the same machine(1), where users connect through ssh to submit jobs.
I would like to restrict the resource allocation (cpu/mem) for all users only inside their ssh session, so that they will not run heavy computations outside the scheduling system (but they can inside jobs).
I know I can use cgroups to implement such limit, e.g.:
cgcreate -g memory,cpu:cpulimited cgset -r cpu.shares=100 cpulimited # limit to ~10% of cpu use cgset -r memory.limit_in_bytes=$((10*1024*1024*1024)) # limit to 10 GB
How can I ensure that any ssh session will be run in this cgroup
cpulimited? but in the same time, when slurm lauches a job, I don't want to bypass the limit set by the scheduler (for any user).
(1) I need a workload manager because I have several users and it can be chaotic without it, however I don't have any other machine that could be the scheduling server (like on any standard cluster).