I installed CUDA toolkit on my computer and started BOINC project on GPU. In BOINC I can see that it is running on GPU, but is there a tool that can show me more details about that what is running on GPU - GPU usage and memory usage?

13 Answers 13

up vote 172 down vote accepted

For Nvidia GPUs there is a tool nvidia-smi that can show memory usage, GPU utilization and temperature of GPU. There also is a list of compute processes and few more options but my graphic card (GeForce 9600 GT) is not fully supported.

Sun May 13 20:02:49 2012       
+------------------------------------------------------+                       
| NVIDIA-SMI 3.295.40   Driver Version: 295.40         |                       
|-------------------------------+----------------------+----------------------+
| Nb.  Name                     | Bus Id        Disp.  | Volatile ECC SB / DB |
| Fan   Temp   Power Usage /Cap | Memory Usage         | GPU Util. Compute M. |
|===============================+======================+======================|
| 0.  GeForce 9600 GT           | 0000:01:00.0  N/A    |       N/A        N/A |
|   0%   51 C  N/A   N/A /  N/A |  90%  459MB /  511MB |  N/A      Default    |
|-------------------------------+----------------------+----------------------|
| Compute processes:                                               GPU Memory |
|  GPU  PID     Process name                                       Usage      |
|=============================================================================|
|  0.           Not Supported                                                 |
+-----------------------------------------------------------------------------+

For linux, use nvidia-smi -l 1 will continually give you the gpu usage info, with in refresh interval of 1 second.

  • 59
    I prefer to use watch -n 1 nvidia-smi to obtain continuous updates without filling the terminal with output – ali_m Jan 27 '16 at 23:59
  • Using watch means your starting a new process every second to poll the cards. Better to do -l, and not every second, I'd suggest every minute or every 5 minutes. – Mick T Apr 19 at 15:55

For Intel GPU's there exists the intel-gpu-tools from http://intellinuxgraphics.org/ project, which brings the command intel_gpu_top (amongst other things). It is similar to top and htop, but specifically for the Intel GPU.

   render busy:  18%: ███▋                                   render space: 39/131072
bitstream busy:   0%:                                     bitstream space: 0/131072
  blitter busy:  28%: █████▋                                blitter space: 28/131072

          task  percent busy
           GAM:  33%: ██████▋                 vert fetch: 0 (0/sec)
          GAFS:   3%: ▋                       prim fetch: 0 (0/sec)
            VS:   0%:                      VS invocations: 559188 (150/sec)
            SF:   0%:                      GS invocations: 0 (0/sec)
            VF:   0%:                           GS prims: 0 (0/sec)
            DS:   0%:                      CL invocations: 186396 (50/sec)
            CL:   0%:                           CL prims: 186396 (50/sec)
           SOL:   0%:                      PS invocations: 8191776208 (38576436/sec)
            GS:   0%:                      PS depth pass: 8158502721 (38487525/sec)
            HS:   0%:                      
            TE:   0%:                      
          GAFM:   0%:                      
           SVG:   0%:                      

Recently I have written a simple command-line utility called gpustat (which is a wrapper of nvidia-smi) : please take a look at https://github.com/wookayin/gpustat.

nvidia-smi does not work on some linux machines (returns N/A for many properties). You can use nvidia-settings instead (this is also what mat kelcey used in his python script).

nvidia-settings -q GPUUtilization -q useddedicatedgpumemory

You can also use:

watch -n0.1 "nvidia-settings -q GPUUtilization -q useddedicatedgpumemory"

for continuous monitoring.

  • 4
    Glad this wasn't a comment. It's exactly what I was searching for when I came across this question. – Score_Under Jun 20 '15 at 0:24
  • Thanks, this is what worked for me, since I have a GeForce card which is not supported by nvidia-smi. – alexg Dec 22 '15 at 9:23
  • 4
    You can do nvidia-settings -q all to see what other parameters you can monitor. I'm monitoring GPUCurrentProcessorClockFreqs and GPUCurrentClockFreqs. – alexg Dec 22 '15 at 9:34
  • 1
    Thanks man, good idea to query all, since each card may have different strings to monitor! – ruoho ruotsi Feb 2 '16 at 19:08

For completeness, AMD has two options:

  1. fglrx (closed source drivers).

    $ aticonfig --odgc --odgt
    
  2. mesa (open source drivers), you can use RadeonTop.

    View your GPU utilization, both for the total activity percent and individual blocks.

I have a GeForce 1060 GTX video card and I found that the following command give me info about card utilization, temperature, fan speed and power consumption:

$ nvidia-smi --format=csv --query-gpu=power.draw,utilization.gpu,fan.speed,temperature.gpu

You can see list of all query options with:

$ nvidia-smi --help-query-gpu
  • It would be worth adding memory.used or (memory.free) as well. – Zoltan Sep 23 at 9:10

For Linux, I use this HTOP like tool that I wrote myself. It monitors and gives an overview of the GPU temperature as well as the core / VRAM / PCI-E & memory bus usage. It does not monitor what's running on the GPU though.

gmonitor

enter image description here

  • 1
    nvidia-settings requires a running X11, which is not always the case. – Victor Sergienko Jul 8 '17 at 0:57
  • works for me with no hassle! – Hennadii Madan Jul 14 '17 at 6:29

For OS X

Including Mountain Lion

iStat Menus

Excluding Mountain Lion

atMonitor

The last version of atMonitor to support GPU related features is atMonitor 2.7.1.

– and the link to 2.7.1 delivers 2.7b.

For the more recent version of the app, atMonitor - FAQ explains:

To make atMonitor compatible with MacOS 10.8 we have removed all GPU related features.

I experimented with 2.7b a.k.a. 2.7.1 on Mountain Lion with a MacBookPro5,2 with NVIDIA GeForce 9600M GT. The app ran for a few seconds before quitting, it showed temperature but not usage:

                                                  screenshot of atMonitor 2.7b on Mountain Lion

Glances has a plugin which shows GPU utilization and memory usage.

enter image description here

http://glances.readthedocs.io/en/stable/aoa/gpu.html

Uses the nvidia-ml-py3 library: https://pypi.python.org/pypi/nvidia-ml-py3

for nvidia on linux i use the following python script which uses an optional delay and repeat like iostat and vmstat

https://gist.github.com/matpalm/9c0c7c6a6f3681a0d39d

$ gpu_stat.py 1 2
{"util":{"PCIe":"0", "memory":"10", "video":"0", "graphics":"11"}, "used_mem":"161", "time": 1424839016}
{"util":{"PCIe":"0", "memory":"10", "video":"0", "graphics":"9"}, "used_mem":"161", "time":1424839018}

The following function appends information such as PID, user name, CPU usage, memory usage, GPU memory usage, program arguments and run time of processes that are being run on the GPU, to the output of nvidia-smi:

function better-nvidia-smi () {
    nvidia-smi
    join -1 1 -2 3 \
        <(nvidia-smi --query-compute-apps=pid,used_memory \
                     --format=csv \
          | sed "s/ //g" | sed "s/,/ /g" \
          | awk 'NR<=1 {print toupper($0)} NR>1 {print $0}' \
          | sed "/\[NotSupported\]/d" \
          | awk 'NR<=1{print $0;next}{print $0| "sort -k1"}') \
        <(ps -a -o user,pgrp,pid,pcpu,pmem,time,command \
          | awk 'NR<=1{print $0;next}{print $0| "sort -k3"}') \
        | column -t
}

Example output:

$ better-nvidia-smi
Fri Sep 29 16:52:58 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 378.13                 Driver Version: 378.13                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GT 730      Off  | 0000:01:00.0     N/A |                  N/A |
| 32%   49C    P8    N/A /  N/A |    872MiB /   976MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+
|   1  Graphics Device     Off  | 0000:06:00.0     Off |                  N/A |
| 23%   35C    P8    17W / 250W |    199MiB / 11172MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0                  Not Supported                                         |
|    1      5113    C   python                                         187MiB |
+-----------------------------------------------------------------------------+
PID   USED_GPU_MEMORY[MIB]  USER    PGRP  %CPU  %MEM   TIME      COMMAND
9178  187MiB                tmborn  9175  129   2.6    04:32:19  ../path/to/python script.py args 42
  • Carefull, I don't think the pmem given by ps takes into account the total memory of the GPU but that of the CPU because ps is not "Nvidia GPU" aware – SebMa May 29 at 14:09

I have had processes terminate (probably killed or crashed) and continue to use resources, but were not listed in nvidia-smi. Usually these processes were just taking gpu memory.

If you think you have a process using resources on a GPU and it is not being shown in nvidia-smi, you can try running this command to double check. It will show you which processes are using your GPUs.

sudo fuser -v /dev/nvidia*

This works on EL7, Ubuntu or other distributions might have their nvidia devices listed under another name/location.

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.