1

I'm working with a Centos Cluster, sharing Nvidia GPUs with other people. Each node on the cluster has 4 GPUs. I'm hoping to find a way to periodically monitor the 4 GPUs and start a (bash/python) script whenever a GPU has more free RAM than a pre-defined threshold.

I'd imagine that it should be possible to parse the output of nvidia-smi but this sounds rather complicated to me.

Does anyone know how to approach this?

4
  • What are you using to schedule jobs on the cluster? Depending on the scheduler, you can just have the jobs set limits on GPU memory, and let the scheduler start jobs when other jobs complete.
    – jsbillings
    Dec 1, 2019 at 17:45
  • I can start jobs interactively / directly from the bash terminal.
    – matohak
    Dec 2, 2019 at 13:38
  • @matohak Hi, did you find a way for this ? (other than using cluster management system like slurm ?)
    – zwlayer
    Mar 9, 2020 at 19:49
  • 1
    I've put this idea aside but I know it'd be easy to use GPUtils in python to monitor GPU memory and launch script: import GPUtil; gpus = GPUtil.getGPUs(); if gpus[0].memoryFree > 6000: launchscript
    – matohak
    Mar 10, 2020 at 16:13

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.