I have logged the GPU utilization of my 3 GPUs (GeForce GTX Titan X) for 5 days with nvidia-smi, every 10 seconds. The GPUs were all being utilized: I count 1307 occurrences of 99 % while only 8 occurrences of 100 %. Why nvidia-smi's GPU so rarely reaches 100%?

enter image description here

enter image description here

Another example from https://openai.com/blog/infrastructure-for-deep-learning/:

enter image description here

  • I haven't found any solution so far. Commented Mar 28, 2016 at 18:24
  • So I am still interested if someone has any idea. Commented Mar 28, 2016 at 18:24
  • It highly depends on what your computations/datasets are, and the load-balancing between the GPUs. Not to mention your hardware which could be a factor. But it seems a little bit curious to want to have exactly 100% of used RAM.
    – Paradox
    Commented Apr 3, 2019 at 11:55
  • @Paradox 100% GPU utilization, not RAM. Commented Apr 4, 2019 at 0:33
  • @FrankDernoucourt Yes, you are right, my bad. But still, RAM or GPU utilization, it's the same: having exactly 100% is difficult unless everything is optimized to the very last bit and purely theoretical. When seeing 98 or 99%, it already seems pretty impressive for multiple components on a single machine. That's why your numbers do not seem explicitly showing a problem to me, depending on what's running on your GPUs.
    – Paradox
    Commented Apr 4, 2019 at 9:20


You must log in to answer this question.

Browse other questions tagged .