I'm working with a Centos Cluster, sharing Nvidia GPUs with other people. Each node on the cluster has 4 GPUs. I'm hoping to find a way to periodically monitor the 4 GPUs and start a (bash/python) script whenever a GPU has more free RAM than a pre-defined threshold.
I'd imagine that it should be possible to parse the output of
nvidia-smi but this sounds rather complicated to me.
Does anyone know how to approach this?