6

Given how everything is a file in Linux, processes don't magically get their information out of nowhere. If you're seeing information without sudo, then that means there's a file somewhere in sysfs, proc, or devfs that can be accessed as a user. Otherwise it could also just be from the kernel ABI (syscalls) rather than a file, where information could also be gathered from functions like ioctl() and socket() if not other direct syscalls like getcpu().

So with that in mind, I'm looking for the source files nvidia-smi reads for its information

In fact, I'd like to extend this further than Nvidia alone since nouveau, radeontop, AMDGPU, and whatever Intel uses, also have similar ways of getting such information that are untold for some reason...

Which files do these processes get their information from?

My ultimate goal is to read these files with Conky and display graphs for them 16 times a second for every GPU in my system, just like I'm doing for my system CPU and RAM use.

Improve this question
2
  • 1
    Not everything is a file in Linux. That is a fairly convenient model for many user applications, via /proc and /sys. However, there are generally also diagnostic routes available for hardware monitoring and setup.
    – Paul_Pedant
    Commented Jan 9 at 19:36
  • yeah that's where I wanted to briefly point out the ABI, since that's really the only other place where you can get info from things like ioctl() or socket(), including things like i2c and all that. :P
    – Tcll
    Commented Jan 9 at 22:12

1 Answer 1

Reset to default
11

You can find this out yourself, with a single command.

strace -o results.txt -e trace=openat nvidia-smi

will run nvidia-smi and create a list of every file it opens as results.txt.

You will find that it uses libnvidia-ml.so.1, which is the NVIDIA Management Library. After loading that library, nvidia-smi opens files like:

  • /dev/nvidia0
  • /dev/nvidia-caps/nvidia-cap2
  • /dev/nvidiactl
  • /dev/nvidia-uvm

You'll find that these device nodes don't produce any useful monitoring information for direct reading; no doubt the library is doing specific ioctl(2) system calls to those devices to get the information it needs. (You can confirm this with strace -o results2.txt -e trace=openat,ioctl nvidia-smi.)

If you wish to do some programming to get the information you want, you probably should read the NVML API Reference Guide and use the libnvidia-ml.so.1 library instead of trying to make raw ioctl() calls in your own code. You'll probably find almost everything you would want to monitor in real time in chapter 2.14, Device Queries.

However, there are some files that look like they might be directly readable:

  • /proc/driver/nvidia/capabilities/mig/config
  • /proc/driver/nvidia/capabilities/mig/monitor
  • /proc/driver/nvidia/params

This gives us hope that /proc/driver/nvidia/ and its sub-directories might be worth examining; alas, there's not much in there that would be worth real-time monitoring.

So, unfortunately it looks like it's not quite as easy as reading some files with Conky; if you aren't willing to do a bit of programming, might I suggest starting with processing the output of

nvidia-smi -q -d MEMORY,UTILIZATION,TEMPERATURE,VOLTAGE,POWER

or a similar command?

And I think running that kind of monitoring in a cycle of 16 times per second is a silly overkill unless you are troubleshooting a specific issue.

Improve this answer
11
  • oh interesting, strace segfaulted for me, so I figured it was nvidia-smi's doing, thanks for letting me know that actually works
    – Tcll
    Commented Jan 9 at 17:50
  • 1
    The NVIDIA ioctl numbers and data formats might be undocumented, so using them without the helper library can be a pain in the backside :-/ If you want to try it, you'll have my sympathy.
    – telcoM
    Commented Jan 9 at 18:09
  • 1
    @Tcll: If you had some computational work that a core could be doing, even interrupting it 100x per second isn't great. Disrupting out-of-order exec is still pretty minor at that rate, with 40M clocks between interruptions vs. an out-of-order window size of a few hundred uops. But worse, context-switch to another process that does some stuff will evict some data from caches and disrupt branch prediction, making things slower for a while switching back.
    – Peter Cordes
    Commented Jan 10 at 2:55
  • 1
    @Tcll: usenix.org/legacy/event/osdi10/tech/full_papers/Soares.pdf measured that disruption just from running kernel code inside a system call, with reduced IPC (insns per cycle) for a while after returning to the interrupted process. e.g. a couple percent degradation even with 500k instructions between pwrite system calls. (40M cycles is maybe like 80M instructions, but you're doing a full context switch which disrupts more, but looking at the numbers it's still minor.) Contention for memory bandwidth and PCIe traffic is still hopefully negligible at 100x per second, though.
    – Peter Cordes
    Commented Jan 10 at 2:55
  • 1
    @Tcll: Hmm, if polling is cheaper than GUI updates, perhaps high-resolution mode could only update the GUI every 4, 8, or 10 samples, so you have more temporal resolution in the graphs while only doing the disruptive heavy lifting less frequently. But each poll still takes a context-switch away from whatever a core could have been doing, so it's not free. I wouldn't want it running on my system 24/7. It also wastes a small amount of power when idle, stopping the core or whole package from going into a deeper sleep if it might otherwise have been able to.
    – Peter Cordes
    Commented Jan 10 at 3:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .