1

I'm working with a Centos Cluster, sharing Nvidia GPUs with other people. Each node on the cluster has 4 GPUs. I'm hoping to find a way to periodically monitor the 4 GPUs and start a (bash/python) script whenever a GPU has more free RAM than a pre-defined threshold.

I'd imagine that it should be possible to parse the output of nvidia-smi but this sounds rather complicated to me.

Does anyone know how to approach this?

Improve this question
4
  • What are you using to schedule jobs on the cluster? Depending on the scheduler, you can just have the jobs set limits on GPU memory, and let the scheduler start jobs when other jobs complete.
    – jsbillings
    Commented Dec 1, 2019 at 17:45
  • I can start jobs interactively / directly from the bash terminal.
    – matohak
    Commented Dec 2, 2019 at 13:38
  • @matohak Hi, did you find a way for this ? (other than using cluster management system like slurm ?)
    – zwlayer
    Commented Mar 9, 2020 at 19:49
  • 1
    I've put this idea aside but I know it'd be easy to use GPUtils in python to monitor GPU memory and launch script: import GPUtil; gpus = GPUtil.getGPUs(); if gpus[0].memoryFree > 6000: launchscript
    – matohak
    Commented Mar 10, 2020 at 16:13

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .