2

Since CUDA 10.1, users are required to have sudo privileges to collect advanced metrics with cuda profiling tools such as nvprof or nsight compute (ncu).

Alternatives to get around it are described here:

The link above mentions that CAP_SYS_ADMIN can be used in order to enable the collection of these metrics.

Trying to understand the problem, I found this insightful stack overflow response:

Correct me If I am wrong, but in order to proceed with CAP_SYS_ADMIN path I should enable the capability for the application AND the user (if it is a non-root user).

I am not familiar with linux capabilities and I am not sure if is best to grant CAP_SYS_ADMIN to user/application or just give the user SUDO access. Why one is better than the other?


EDIT: As of right now, I am still unable to get it to work.

# First I executed
$ sudo setcap cap_sys_admin+ep /usr/local/cuda/bin/nvprof
# This is the command that I am executing after installing the CUDA toolkit 10.2.
$ /usr/local/cuda/bin/nvprof -o output-detailed.nvvp -f --analysis-metrics /usr/local/cuda/extras/demo_suite/vectorAdd

[Vector addition of 50000 elements]
==142443== NVPROF is profiling process 142443, command: /usr/local/cuda/extras/demo_suite/vectorAdd
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
==142443== Some kernel(s) will be replayed on device 0 in order to collect all events/metrics.
==142443== Warning: ERR_NVGPUCTRPERM - The user does not have permission to profile on the target device. See the following link for instructions to enable permissions and get more information: https://developer.nvidia.com/ERR_NVGPUCTRPERM
Failed to launch vectorAdd kernel (error code unknown error)!
==142443== Warning: ERR_NVGPUCTRPERM - The user does not have permission to profile on the target device. See the following link for instructions to enable permissions and get more information: https://developer.nvidia.com/ERR_NVGPUCTRPERM
==142443== Warning: Some profiling data are not recorded. Make sure cudaProfilerStop() or cuProfilerStop() is called before application exit to flush profile data.
==142443== Generated result file: /results/nvprof/output-detailed.nvvp

But it only works if I run with sudo.

$ sudo /usr/local/cuda/bin/nvprof -o output-detailed.nvvp -f --analysis-metrics /usr/local/cuda/extras/demo_suite/vectorAdd

[Vector addition of 50000 elements]
==142687== NVPROF is profiling process 142687, command: /usr/local/cuda/extras/demo_suite/vectorAdd
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
==142687== Some kernel(s) will be replayed on device 0 in order to collect all events/metrics.
Replaying kernel "vectorAdd(float const *, float const *, float*, int)" (done)
Copy output data from the CUDA device to the host memory
Test PASSED
Done
==142687== Generated result file: /home/agostini/Development/nvprof/output-detailed.nvvp

Why it is not enough to grant the capabilities to the executable and have super user to run the application without sudo. Is PAM setup really needed even for a user in the sudo group?

1
  • The Principle of Least Privilege suggests that granting root should always be the option of last resort. Any working alternative is superior.
    – Tom
    Commented Aug 11, 2022 at 2:53

1 Answer 1

1

I don't know exactly how CAP_SYS_ADMIN works, but it's probably much easier to just follow the instructions that make it so a non root user can use the profiler.

echo 'options nvidia "NVreg_RestrictProfilingToAdminUsers=0"' | sudo tee -a /etc/modprobe.d/nvidia.conf
sudo update-initramfs -u 
sudo reboot

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .