I have logged the GPU utilization of my 3 GPUs (GeForce GTX Titan X) for 5 days with
nvidia-smi, every 10 seconds. The GPUs were all being utilized: I count 1307 occurrences of
99 % while only 8 occurrences of
100 %. Why nvidia-smi's GPU so rarely reaches 100%?
Another example from https://openai.com/blog/infrastructure-for-deep-learning/: