2

I have logged the GPU utilization of my 3 GPUs (GeForce GTX Titan X) for 5 days with nvidia-smi, every 10 seconds. The GPUs were all being utilized: I count 1307 occurrences of 99 % while only 8 occurrences of 100 %. Why nvidia-smi's GPU so rarely reaches 100%?

enter image description here

enter image description here

Another example from https://openai.com/blog/infrastructure-for-deep-learning/:

enter image description here

  • I haven't found any solution so far. – Franck Dernoncourt Mar 28 '16 at 18:24
  • So I am still interested if someone has any idea. – Franck Dernoncourt Mar 28 '16 at 18:24
  • It highly depends on what your computations/datasets are, and the load-balancing between the GPUs. Not to mention your hardware which could be a factor. But it seems a little bit curious to want to have exactly 100% of used RAM. – Paradox Apr 3 at 11:55
  • @Paradox 100% GPU utilization, not RAM. – Franck Dernoncourt Apr 4 at 0:33
  • @FrankDernoucourt Yes, you are right, my bad. But still, RAM or GPU utilization, it's the same: having exactly 100% is difficult unless everything is optimized to the very last bit and purely theoretical. When seeing 98 or 99%, it already seems pretty impressive for multiple components on a single machine. That's why your numbers do not seem explicitly showing a problem to me, depending on what's running on your GPUs. – Paradox Apr 4 at 9:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.