Slurm on desktop computers, how to prioritize the owner

Question

In short: On a Slurm cluster, I need some computers to be available and responsive to their respective owners during work hours.

Problem: I manage a small (but growing) heterogeneous cluster with around 10 nodes, where some of the nodes are not dedicated. These are desktop computers used by colleagues on the same network during work hours and they would prefer to work on responsive machines. During nights and weekends however, we pool all of our computers and some dedicated nodes together for batch jobs.

I recently switched from HT Condor to Slurm because it fits our needs better in all except one aspect: prioritizing the owner of the machine for regular work not related to the cluster. On Condor a node could be configured to suspend, preempt or kill jobs depending on criteria such as

• Time of day or weekday (machines are used during the day on weekdays)
• Keyboard activity (some users may be working late)
• CPU activity from processes other than those spawned by the cluster (users may leave some of their own processes running overnight, that should run without interference)

I would like to mimic any of these behaviors when using Slurm, or find a way to not bother the owner using the computer.

Additional info: All the nodes use Ubuntu 18.04-19.04 with slurm found in apt, i.e. version 18+. The cluster uses cgroups for limit enforcement and is configured to use cores as consumable resource, as in

SelectType=select/cons_res
SelectTypeParameters=CR_Core

I do not have sudo-rights on most desktop computers, so either I need a "set and forget" solution that one time when I configure my colleagues PC, or something I can do from the head node where I do have sudo.

Attempts: I have considered these options but remain unsatisfied:

• For time of day/weekday, use either crontab or systemd with OnCalendar events in slurmd.service, to either:
  • start/stop the daemon. This may be the easiest way but kills jobs in a non-clean way.
  • launch a script that sets the node state using scontrol to down/resume/drain/etc, possibly from head node. I haven't tried this one as I can't figure out how to do this outside of the interactive mode of scontrol.
• For responsiveness, use "systemd edit slurmd.service" to add resource control by setting CpuWeight=5 under [Service]. This should prioritize every other process, but doesn't seem to work as I intended because the jobs make the computers sluggish anyway. I thought jobs would be subprocesses of slurmd and be subject to the same CpuWeight. If this actually worked well, it could solve the whole problem.

I feel there should be a better way to achieve what I want. Any help is appreciated.

Answer 1

After a couple of days I managed to answer my own question. In hindsight it was quite simple.

Responsiveness: The slurmd daemon can be started with command line arguments, list them with slurmd -h. In particular , slurmd -n 19 sets the highest nice-value (and thus lowest priority) for the daemon and all its subprocesses. On desktop computers, I simply

• edited /etc/systemd/system/slurmd.service appending -n 19 to ExecStart, i.e. ExecStart=/usr/local/sbin/slurmd $SLURMD_OPTIONS -n 19
• reloaded systemd daemons with systemctl daemon-reload
• restarted the slurmd daemon, systemctl restart slurmd.service

Memory reservation: Some memory can be reserved to the system. I leave 8GB to the owner by adding MemSpecLimit=8000 to the node specifications in slurmd.conf. To actually enforce memory limits there were some additional steps:

• Select Core and Memory as consumable resources, by setting SelectTypeParameters=CR_Core_Memory in slurmd.conf.

• Add cgroups task plugin by setting TaskPlugin=task/affinity,task/cgroup in slurmd.conf and then setting ConstrainRAMSpace=yes in cgroup.conf.

• Because we are on Ubuntu, enable memory and swap cgroups by adding the line GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1" to /etc/default/grub

Work hours on weekdays Some of my colleagues would like zero distraction during work hours. This is easy to do with scontrol from the head node, to set their node state to "down" during work hours, and to "resume" after work hours. I do this automatically with systemd timers:

• First, make an executable script that updates the node states of the desktops in question using scontrol:

#!/bin/bash
# slurm-update.sh - Updates the state on nodes belonging to the work-hour desktops partition.
systemctl start slurmd
for node in $(sinfo -h --partition=WHdesktops --format="%n"); do
        state=$(sinfo -h --node=$node --format="%T")
        echo "Setting node $node to state=$1 with reason=$2"
        scontrol update NodeName=$node state=$1 reason="$2" || echo "State on $node is already $(sinfo -h --node=$node --format=\"%T\")"
done

This takes two arguments, the new state and a reason for it.

• Create a service/timer pair of files in the directory /etc/systemd/system to run the script above at certain times. Make one pair per state you want to set (for instance I made 3 pairs, to set down, drain and resume). The pair for setting "down" looks like this

# /etc/systemd/system/slurm-down.service:
[Unit]
Description=Shut down all SLURM desktop nodes
[Service]
Type=simple
ExecStart=/bin/bash /mnt/nfs/slurm_fs/systemd/slurm-update.sh down afterhours
StandardError=journal

and

# /etc/systemd/system/slurm-down.timer:
[Unit]
Description=Timer for shutting down slurm on desktop nodes on weekdays
[Timer]
Unit=slurm-down.service
OnBootSec=10min
# Run hourly on weekdays between 8:05 to 18:05
OnCalendar=Mon..Fri *-*-* 8..18:05:00
[Install]
WantedBy=multi-user.target

• Reload the daemons with systemctl daemon-reload and then enable and start the timer only, not the service: systemctl enable --now slurm-down.timer.

• Repeat the steps for the resume state after work hours, and optionally a drain state an hour or so before the down state.