0

I am trying something odd. I have a slurm cluster configured that has 4 compute nodes. 2 of which are Windows 11 machines running WSL2 and I have it working.

right now, I am trying to add GPU support to the SLURM cluster. For the 2 compute nodes that are native ubuntu 22.04 systems, the nvidia devices are showing as /dev/nvidia[0-3] but for the WSL2 (Ubuntu 22.04) there is nothing in /dev/ that speak to gpus.

The hardware on the 2 PCs are different, but they are both running Windows 11 with Nvidia Driver 537.13, WSL2 Ubuntu 22.04, and cuda drivers installed as prescribed here: NVidia Site

the output of nvidia-smi on both machines:

Thu Sep  7 23:28:30 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.103                Driver Version: 537.13       CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3090        On  | 00000000:01:00.0 Off |                  N/A |
|  0%   24C    P8              11W / 420W |     53MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 3090        On  | 00000000:2E:00.0 Off |                  N/A |
|  0%   24C    P8              14W / 420W |      0MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   2  NVIDIA GeForce RTX 3090        On  | 00000000:41:00.0  On |                  N/A |
|  0%   25C    P5              43W / 420W |   2753MiB / 24576MiB |      2%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   3  NVIDIA GeForce RTX 3090        On  | 00000000:61:00.0 Off |                  N/A |
|  0%   25C    P8              12W / 420W |      0MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A        22      G   /Xwayland                                 N/A      |
|    0   N/A  N/A        22      G   /Xwayland                                 N/A      |
|    0   N/A  N/A        23      G   /Xwayland                                 N/A      |
|    1   N/A  N/A        22      G   /Xwayland                                 N/A      |
|    1   N/A  N/A        22      G   /Xwayland                                 N/A      |
|    1   N/A  N/A        23      G   /Xwayland                                 N/A      |
|    2   N/A  N/A        22      G   /Xwayland                                 N/A      |
|    2   N/A  N/A        22      G   /Xwayland                                 N/A      |
|    2   N/A  N/A        23      G   /Xwayland                                 N/A      |
|    3   N/A  N/A        22      G   /Xwayland                                 N/A      |
|    3   N/A  N/A        22      G   /Xwayland                                 N/A      |
|    3   N/A  N/A        23      G   /Xwayland                                 N/A      |
+---------------------------------------------------------------------------------------+
Thu Sep  7 23:47:19 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.103                Driver Version: 537.13       CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 2080 Ti     On  | 00000000:01:00.0 Off |                  N/A |
|  0%   25C    P8              29W / 260W |    433MiB / 11264MiB |      1%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A        23      G   /Xwayland                                 N/A      |
+---------------------------------------------------------------------------------------+

Both machines ls /dev/:

autofs           fuse          loop0   ptmx        ram5    stderr  tty19  tty32  tty46  tty6     vcs2   vcsu3
block            hugepages     loop1   ptp0        ram6    stdin   tty2   tty33  tty47  tty60    vcs3   vcsu4
bsg              hvc0          loop2   ptp_hyperv  ram7    stdout  tty20  tty34  tty48  tty61    vcs4   vcsu5
btrfs-control    hvc1          loop3   pts         ram8    tty     tty21  tty35  tty49  tty62    vcs5   vcsu6
bus              hvc2          loop4   ram0        ram9    tty0    tty22  tty36  tty5   tty63    vcs6   vfio
char             hvc3          loop5   ram1        random  tty1    tty23  tty37  tty50  tty7     vcsa   vhost-net
console          hvc4          loop6   ram10       rtc     tty10   tty24  tty38  tty51  tty8     vcsa1  virtio-ports
core             hvc5          loop7   ram11       rtc0    tty11   tty25  tty39  tty52  tty9     vcsa2  vport0p0
cpu_dma_latency  hvc6          mapper  ram12       sda     tty12   tty26  tty4   tty53  ttyS0    vcsa3  vport0p1
cuse             hvc7          mem     ram13       sdb     tty13   tty27  tty40  tty54  ttyS1    vcsa4  vsock
disk             initctl       mqueue  ram14       sdc     tty14   tty28  tty41  tty55  ttyS2    vcsa5  zero
dri              kmsg          net     ram15       sg0     tty15   tty29  tty42  tty56  ttyS3    vcsa6
dxg              kvm           null    ram2        sg1     tty16   tty3   tty43  tty57  urandom  vcsu
fd               log           nvram   ram3        sg2     tty17   tty30  tty44  tty58  vcs      vcsu1
full             loop-control  ppp     ram4        shm     tty18   tty31  tty45  tty59  vcs1     vcsu2

Ive tried reinstalling cuda on both windows, and cuda in wsl2 (as prescribed above) and googling the issue.

My question is "where can I map the nvidia devices like I would in /dev/nvidia0 on traditional ubuntu installs? in gres.conf for SLURM"

1 Answer 1

0

Apparently DGX is the name of your graphics cards in wsl Linux services:

  photoprism:
    environment:
      ...
    devices:
      - "/dev/dxg:/dev/nvidia0"
    ...
container: host etc

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .