In short: On a Slurm cluster, I need some computers to be available and responsive to their respective owners during work hours.
Problem: I manage a small (but growing) heterogeneous cluster with around 10 nodes, where some of the nodes are not dedicated. These are desktop computers used by colleagues on the same network during work hours and they would prefer to work on responsive machines. During nights and weekends however, we pool all of our computers and some dedicated nodes together for batch jobs.
I recently switched from HT Condor to Slurm because it fits our needs better in all except one aspect: prioritizing the owner of the machine for regular work not related to the cluster. On Condor a node could be configured to suspend, preempt or kill jobs depending on criteria such as
- Time of day or weekday (machines are used during the day on weekdays)
- Keyboard activity (some users may be working late)
- CPU activity from processes other than those spawned by the cluster (users may leave some of their own processes running overnight, that should run without interference)
I would like to mimic any of these behaviors when using Slurm, or find a way to not bother the owner using the computer.
Additional info: All the nodes use Ubuntu 18.04-19.04 with slurm found in apt, i.e. version 18+. The cluster uses cgroups for limit enforcement and is configured to use cores as consumable resource, as in
I do not have sudo-rights on most desktop computers, so either I need a "set and forget" solution that one time when I configure my colleagues PC, or something I can do from the head node where I do have sudo.
Attempts: I have considered these options but remain unsatisfied:
- For time of day/weekday, use either crontab or systemd with OnCalendar events in slurmd.service, to either:
- start/stop the daemon. This may be the easiest way but kills jobs in a non-clean way.
- launch a script that sets the node state using scontrol to down/resume/drain/etc, possibly from head node. I haven't tried this one as I can't figure out how to do this outside of the interactive mode of scontrol.
- For responsiveness, use "systemd edit slurmd.service" to add resource control by setting CpuWeight=5 under [Service]. This should prioritize every other process, but doesn't seem to work as I intended because the jobs make the computers sluggish anyway. I thought jobs would be subprocesses of slurmd and be subject to the same CpuWeight. If this actually worked well, it could solve the whole problem.
I feel there should be a better way to achieve what I want. Any help is appreciated.