1

I build a docker container FROM nvidia/cuda:8.0-devel-ubuntu16.04 in my Dockerfile to have the CUDA Toolkit installed.

My architecture is the one depicted in the official nvidia-docker repo

nvidia-docker layers

After the build and run I get

$ nvidia-smi
bash: nvidia-smi: command not found

I have a DOCKER_HOST that points to the running Docker Nvidia container (the GPUs machine) like

export DOCKER_HOST=tcp://x.x.x.x:2376
export DOCKER_TLS_VERIFY=1
set NVIDIA_VER=367.57

and the I connect to the docker instance binding the ports when doing the tunnel to the machine:

$ ssh -i "$DOCKER_CERT" docker@$IP -g -R 10250:localhost:10250 -L 0.0.0.0:3000:127.0.0.1:3000  -L 0.0.0.0:8181:127.0.0.1:8181 -L 5858:127.0.0.1:5858 -L 4567:127.0.0.1:4567

My docker instance is started as usual

$ docker run --rm -it --name $CONTAINER_NAME -p 3000:3000 $CONTAINER_IMG:$CONTAINER_VERSION $CMD

I can connect to the docker nvidia instance from the docker host:

loreto@nvidia-docker:~$ sudo nvidia-docker run --rm nvidia/cuda nvidia-smi
Fri Mar 17 09:08:03 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 367.57                 Driver Version: 367.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GRID K520           Off  | 0000:00:03.0     Off |                  N/A |
| N/A   35C    P8    17W / 125W |      0MiB /  4036MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

and even

loreto@nvidia-docker:~$ nvidia-smi
Fri Mar 17 09:12:59 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 367.57                 Driver Version: 367.57                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GRID K520           Off  | 0000:00:03.0     Off |                  N/A |
| N/A   35C    P8    17W / 125W |      0MiB /  4036MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |

while from my container $CONTAINER_NAME when running I cannot see it

$ docker exec -it $CONTAINER_NAME bash
$ nvidia-smi
nvidia-smi: command not found

The $CONTAINER_NAME was built FROM nvidia/cuda:8.0-devel-ubuntu16.04

I have asked this question as issue to nvidia-docker github repo here.

[UPDATE] I have solved the issue, attaching the devices and setting the driver on the container when running it like:

docker run --rm -it --device=/dev/nvidiactl --device=/dev/nvidia-uvm --device=/dev/nvidia0 -v nvidia_driver_367.57:/usr/local/nvidia:ro --name $CONTAINER_NAME -p 3000:3000 $CONTAINER_IMG:$CONTAINER_VERSION $CMD

Of course the DOCKER_HOST must be set and the tunnel must be open.

3
  • 1
    I think you may be confused about the usage of docker vs. nvidia-docker. If you want the benefits of nvidia docker, you need to start the container with nvidia-docker. The reason for this is discussed here: "The required character devices and driver files are mounted when starting the container on the target machine" If you start the container with docker, that won't happen. So to demonstrate running nvidia-smi from within the container, I suggest sudo nvidia-docker run --rm -ti nvidia/cuda bash then nvidia-smi Commented Mar 20, 2017 at 17:38
  • @RobertCrovella uhm things are becoming clear...in the meanwhile I try to build the container, is this the same of running my container passing the nvidia device/volume i.e. like docker run -it --device=/dev/nvidiactl --device=/dev/nvidia-uvm --device=/dev/nvidia0 --volume-driver nvidia-docker -v nvidia_driver_367.57:/usr/local/nvidia:ro --name $CONTAINER_NAME -p 3000:3000 $CONTAINER_IMG:$CONTAINER_VERSION bash? Commented Mar 20, 2017 at 17:43
  • 1
    I think it's unlikely we're going to sort this out in the comments. My suggestion would be to get familiar with nvidia-docker by following the tutorial here. There is a particular point in that tutorial where you start a bash shell in the container. If you want to demonstrate nvidia-smi, just run it at that point. Commented Mar 20, 2017 at 17:50

1 Answer 1

2

Install nvidia-docker over normal docker installed version.

Command is:

curl https://get.docker.com | sh
&& sudo systemctl --now enable docker

Then

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
&& curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
&& curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list |
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' |
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Then

sudo apt-get update

Then

sudo apt-get install -y nvidia-docker2

Now once you have installed nvidia-docker2 - First check the groups your user id is part of using the groups command

groups

If you don't see docker as a listed group then run this command first

sudo groupadd docker

sudo usermod -aG docker $USER

Now you need to log out and log back again (if this is not possible due to a remote ssh server - then open a new terminal with ssh to same server).

Run

sudo systemctl restart docker

Now check the groups again. You should see your user id having the docker group. If this is not the case - fix this first. Then run:

sudo docker run --rm --gpus all nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi

You should have nvidia-smi running now within a docker

Select the appropriate docker version with the right nvidia-driver version and version of ubuntu to use for your application from this link - https://hub.docker.com/r/nvidia/cuda/tags

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .