12

In my GPU #0, 11341MiB of GPU RAM is used, and no process is listed by nvidia-smi. How is that possible, and how can I get my memory back?

Thu Aug 18 14:27:58 2016
+------------------------------------------------------+
| NVIDIA-SMI 352.63     Driver Version: 352.63         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX TIT...  Off  | 0000:02:00.0     Off |                  N/A |
| 29%   61C    P2    71W / 250W |  11341MiB / 12287MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX TIT...  Off  | 0000:03:00.0     Off |                  N/A |
| 22%   42C    P0    71W / 250W |     23MiB / 12287MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX TIT...  Off  | 0000:82:00.0     Off |                  N/A |
| 22%   35C    P0    69W / 250W |     23MiB / 12287MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX TIT...  Off  | 0000:83:00.0     Off |                  N/A |
|  0%   33C    P0    60W / 250W |     23MiB / 12287MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

I had launched a Theano Python script with a lib.cnmem=0.9 flag, which explains why it used 11341MiB of GPU memory (the CNMeM library is a "simple library to help the Deep Learning frameworks manage CUDA memory.".). However, I killed the script, and was expecting the GPU memory to get released.

  • pkill -9 python and sudo nvidia-smi --gpu-reset -i 0 did not help.
  • sudo fuser -v /dev/nvidia* didn't show any unwanted process.

I use a GeForce GTX Titan Maxwell with Ubuntu 14.04.4 LTS x64.

2
  • [Does this flushing][stackoverflow.com/questions/15197286/… out RAM from cuda works Jun 13, 2020 at 16:23
  • 4
    The memory did not get released because you used SIGKILL(9). The SIGKILL signal forces the process to stop executing immediately. The program cannot ignore this signal. This process does not get to clean-up either. Always use SIGTERM(15) and if that doesn't work use SIGKILL as the last resource. Jul 21, 2020 at 0:12

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.