1

I have resources (e.g. GPU) I can only use once at a time. When performing experiments I always need to know on time of a batch files creation, what experiments I want to execute in the future. But I am a person who likes to change its mind, but I hate to quit jobs.

Is there something like Slurm, that runs only on one machine, which I can pass jobs to a queue and remove them if necessary?

I am looking for applications that either work on GPU load (in contrast to batch/at that works on CPU load) or executes the jobs sequentially. This means that only one job runs at a time on one resource (GPU) and the next one starts when the execution of the previous job finished. I also want to be able to manage the queue in order to give jobs a higher priority or delete them.

  • batch? It's very simple, and nothing like SLURM, but it'll be installed already. – Kusalananda Feb 1 at 14:22
  • with atd (batch etc.) I can only set a load threashold for the cpu, but not for the gpu. Furthermore I found no solution for executing a queue sequentialy (Job 1 finishes, Job 2 starts, ...) – Martin Feb 4 at 12:58
  • For sequential jobs, submit them as one single job. No, you can probably not get batch to care about the GPU, only the general system load. – Kusalananda Feb 4 at 13:00
  • But I dont want to add them as a single job. This does not provide any advantage compared to a batchfile. – Martin Feb 4 at 13:02
  • Hmm... If you want to run jobs sequentially, then why bother with SLURM? – Kusalananda Feb 4 at 13:05
0

Would it be acceptable to have the jobs run through a simple queue manager of your own?

#!/bin/bash

while ! mkdir /tmp/my_gpu_lockdir; do
    sleep $((RANDOM))
done

trap 'rmdir /tmp/my_gnu_lockdir' ERR EXIT

.... your actual task here ...
  • I would prefer that the queuing system takes care of that. Additionally, I can not see weather this is still pending, I can't change the priority and so on.. But a good quick-shot. – Martin Feb 5 at 9:53
  • You could perhaps wrap this with batch to have the job resubmit itself if the lock folder exists, and then use the regular atq / atrm for managing the queue. – tripleee Feb 5 at 9:56
0

I found a solution that perfectly fits my needs. I have the issue, that I have only one GPU, but want to have a queue to add jobs to, see their status and, if needed, delete it from the queue again.

After some research on Google, I found task-spooler (tsp). With this command-line tool, it is fairly easy to add jobs to the queue and follow their results. So far I only use one queue, but it is also scale-able to more.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.