8

We have one beefy ubuntu rig for our research department that can do heavy lifting with its CPUs and GPUs. All of our researchers SSH into the machine and run (machine learning) workloads on the system.

The problem is that we are having collisions with people using the system at the same time, and a simple chatbox where people call dibs hasn't sufficed. Essentially, if researcher A wants to do a time-sensitive benchmark using the GPUs, we don't want anybody else to touch the GPUs in order to maintain validity.

I am wondering if there is a tool available that can schedule and grant users exclusive access to certain commands or devices. All tasks are run via a centralised Conda (python) installation that is accessible via a custom group. Everybody SSHs into the system. Perhaps it would be possible to block SSH access/make GPUs exclusive/block python access?

EDIT: I should have specified earlier that while we do have an active userbase among our research group, we'd prefer to not complicate the setup with a queueing system. A less intrusive (more naive) change to our setup would be heavily preferred. I am sorry for not mentioning this earlier.

4
  • 2
    This sounds very much it's a task for your internal programming team to address. They could ask about specific issues here but I think your question is too broad to be answered usefully on StackExchange. (Just my opinion, mind.)
    – roaima
    Mar 25 at 10:22
  • I should've been more specific - I've added some additional information now Mar 25 at 11:00
  • In the unlikely event there is no tool, you may be able roll your own by modifying group membership with a cronjob (or manually), or similar.
    – Tom
    Mar 25 at 23:41
  • 1
    We have a sister site for such questions. Ask us at Sotware Recommendations, we love to help :-) Mar 26 at 8:38

3 Answers 3

9

The normal way that these things are handled is by using a queuing system and only allowing users to submit jobs through the queue. A commonly used one that I have seen in research institutes I've worked in is TORQUE:

TORQUE is a resource manager providing control over batch jobs and distributed compute nodes. Basically, one can setup a home or small office Linux cluster and queue jobs with this software. A cluster consists of one head node and many compute nodes. The head node runs the torque-server daemon and the compute nodes run the torque-client daemon. The head node also runs a scheduler daemon.

There are several others as well. Talk to your system administrator about setting up a proper queuing system so you can all use the machine effectively. This is the standard setup for accessing shared clusters.

https://help.ubuntu.com/community/TorquePbsHowto

3
  • Thank you for your answer! I should've specified that we'd prefer a more naive change to our rig than an explicit queueing system, but I'll definitely bring your recommendation up. Mar 25 at 10:57
  • 1
    There's nothing in Linux/Unix as standard to do all of this, but it might be straightforward to cobble something together with disabling accounts/changing user groups, timeout and other standard tools in a simple bash script. However from your description it seems they need to have an open session to run the job - which means you are limiting the capacity of the system to working hours - which round here equates to around 25% of the time. Thats a big waste. I suggest you want to block direct access to the service and implement a job queueing system.
    – symcbean
    Mar 25 at 13:41
  • 2
    @TiesRobroek According to the link, TORQUE is not free. You can also consider slurm. Often used in scientific environments and released under GPL2. Your constraints do call for a queuing system with interactive access disallowed.
    – doneal24
    Mar 25 at 18:50
1

My thoughts on it:

  • Disabling simultaneous SSH access doesn't seem right: from my experience people love to open SSH connections and leave them running without having any applications running. Logging in again and again could make many people upset.
  • Disabling simultaneous python access doesn't seem right: people may run tasks outside of accessing your CPU/GPU resources.

Here's how I'd approach it:

  • There must be a way to monitor your CPU/GPU load.
  • You create a task (running under root) which checks these values every second, if they are high enough, you basically chmod 600/700 files_required_to_access_CPU_GPU. Once the load is low or zero, you chmod 644/755 the_same_files.

I could probably program this for you, but I'll need access to the system. I don't see nothing extra complicated.

3
  • 2
    If files are open then changing their permissions will have no effect on those processes already using them. Have I misunderstood you?
    – roaima
    Mar 25 at 11:04
  • 1
    Since the OP is concerned about simultaneous usage, nvidia-smi will report if the GPU is in use. Monitoring load on the GPU doesn't really make sense, only allocation. I'd assume that other GPU models have a similar utility.
    – doneal24
    Mar 25 at 12:19
  • 1
    If files are open then changing their permissions will have no effect on those processes already using them. exactly Mar 25 at 23:37
1

You can prevent new logins (except by root) by creating a /etc/nologin file. This requires root privileges. BUT we want to be careful to always remove this file when the user is finished with the system. That means we should have some privileged process, that creates the file, awaits some signal or closing, then deletes the file. That's a bit complex using sudo since the process runs in a separate privileged context, and we can't send it signals w/o privilege. We can however communicate from user to privilege process over a pipe.

Consider a bash shell script: _mkexclude

#!/bin/bash
MSG="$1 has exclusive access"
XFILE="/etc/nologin"

[ -e "$XFILE" ] &&  echo "Exclusivity already established, FAIL" >&2  && exit 3
# make exclusive
echo ${MSG} > $XFILE

trap '/bin/rm -rf $XFILE; exit' USR1 QUIT HUP EXIT TERM
# wait to read any line before continuing from stdin cleanup
read f

# make non-exclusive
/bin/rm -rf $XFILE

The script MUST be configured in /etc/sudoers.d as a NOPASSWD command for the proper users. (or group). The script checks to see if there is already a nologin file. If not it creates a new one. Then it WAITS on either a read or a signal. If the originating (bash) shell exits, so will the exclusivity.

A decent way to invoke this script is via bash shell functions ...

exclude() { coproc EXCLUDE (sudo _mkexclusive $(id -un) ); }
disexclude() { echo FOO >&${EXCLUDE[1]} ; }

[give this to users in their ~/.bashrc]

Any user, allowed to sudo the script, can can prevent further logins by executing the $ exclude function. Such a user can, ONLY IN THE CONTEXT of the VERY SAME SHELL execute **disexclude** re-allow others to login.

IF a user closes the shell that creates the exclusion, OR logs-out, then the coproc should remove the $XFILE that prevents others to login.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.