0

I build a docker container FROM nvidia/cuda:8.0-devel-ubuntu16.04 in my Dockerfile to have the CUDA Toolkit installed.

My architecture is the one depicted in the official nvidia-docker repo

nvidia-docker layers

After the build and run I get

$ nvidia-smi
bash: nvidia-smi: command not found

I have a DOCKER_HOST that points to the running Docker Nvidia container (the GPUs machine) like

export DOCKER_HOST=tcp://x.x.x.x:2376
export DOCKER_TLS_VERIFY=1
set NVIDIA_VER=367.57

and the I connect to the docker instance binding the ports when doing the tunnel to the machine:

$ ssh -i "$DOCKER_CERT" docker@$IP -g -R 10250:localhost:10250 -L 0.0.0.0:3000:127.0.0.1:3000  -L 0.0.0.0:8181:127.0.0.1:8181 -L 5858:127.0.0.1:5858 -L 4567:127.0.0.1:4567

My docker instance is started as usual

$ docker run --rm -it --name $CONTAINER_NAME -p 3000:3000 $CONTAINER_IMG:$CONTAINER_VERSION $CMD

I can connect to the docker nvidia instance from the docker host:

loreto@nvidia-docker:~$ sudo nvidia-docker run --rm nvidia/cuda nvidia-smi
Fri Mar 17 09:08:03 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 367.57                 Driver Version: 367.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GRID K520           Off  | 0000:00:03.0     Off |                  N/A |
| N/A   35C    P8    17W / 125W |      0MiB /  4036MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

and even

loreto@nvidia-docker:~$ nvidia-smi
Fri Mar 17 09:12:59 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 367.57                 Driver Version: 367.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GRID K520           Off  | 0000:00:03.0     Off |                  N/A |
| N/A   35C    P8    17W / 125W |      0MiB /  4036MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |

while from my container $CONTAINER_NAME when running I cannot see it

$ docker exec -it $CONTAINER_NAME bash
$ nvidia-smi
nvidia-smi: command not found

The $CONTAINER_NAME was built FROM nvidia/cuda:8.0-devel-ubuntu16.04

I have asked this question as issue to nvidia-docker github repo here.

[UPDATE] I have solved the issue, attaching the devices and setting the driver on the container when running it like:

docker run --rm -it --device=/dev/nvidiactl --device=/dev/nvidia-uvm --device=/dev/nvidia0 -v nvidia_driver_367.57:/usr/local/nvidia:ro --name $CONTAINER_NAME -p 3000:3000 $CONTAINER_IMG:$CONTAINER_VERSION $CMD

Of course the DOCKER_HOST must be set and the tunnel must be open.

  • 1
    I think you may be confused about the usage of docker vs. nvidia-docker. If you want the benefits of nvidia docker, you need to start the container with nvidia-docker. The reason for this is discussed here: "The required character devices and driver files are mounted when starting the container on the target machine" If you start the container with docker, that won't happen. So to demonstrate running nvidia-smi from within the container, I suggest sudo nvidia-docker run --rm -ti nvidia/cuda bash then nvidia-smi – Robert Crovella Mar 20 '17 at 17:38
  • @RobertCrovella uhm things are becoming clear...in the meanwhile I try to build the container, is this the same of running my container passing the nvidia device/volume i.e. like docker run -it --device=/dev/nvidiactl --device=/dev/nvidia-uvm --device=/dev/nvidia0 --volume-driver nvidia-docker -v nvidia_driver_367.57:/usr/local/nvidia:ro --name $CONTAINER_NAME -p 3000:3000 $CONTAINER_IMG:$CONTAINER_VERSION bash? – loretoparisi Mar 20 '17 at 17:43
  • 1
    I think it's unlikely we're going to sort this out in the comments. My suggestion would be to get familiar with nvidia-docker by following the tutorial here. There is a particular point in that tutorial where you start a bash shell in the container. If you want to demonstrate nvidia-smi, just run it at that point. – Robert Crovella Mar 20 '17 at 17:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.