In our research group we are running a computing server for deep learning with a number of NVIDIA Titan X graphics cards and quite some CPU cores. Given that it is a research lab and we have ~10 people using the machine the load on the CPU/GPU cores is almost always high.

I am now in charge showing that the machine is overly used and I can propose hardware upgrades. To make an argument, I want to create a detailed history of the CPU/GPU/MEM usage on the machine. The problem is, I don't know the right tools for the job. Of course, I can do some scripting but I prefer off-the-shelf tools since I am not a system administrator :) For monitoring the CPU/GPU usage I typically use nvidia-smi and htop, but these are not suitable for generating long-time histories.

Any recommendations on creating such histories?

  • Do you mean a monitoring tool like munin? – lw1.at Sep 14 '16 at 12:11
  • munin indeed looks like a good solution. In the contrib directory of munin there are some plugins for monitoring GPU utilization/temperature. I will give them a try! github.com/munin-monitoring/contrib/blob/master/plugins/gpu/… – verified.human Sep 19 '16 at 9:45
  • There is also a plugin called nvidia_ (for debian it is in the munin-plugins-core package) which should return temperature, clock speed and voltage. – lw1.at Sep 19 '16 at 11:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.