9

How is it possible to control the fan speed of multiple consumer NVIDIA GPUs such as Titan and 1080 Ti on a headless node running Linux?

|improve this question
5

The following is a simple method that does not require scripting, connecting fake monitors, or fiddling and can be executed over SSH to control multiple NVIDIA GPUs' fans. It has been tested on Arch Linux.

Identify your cards' PCI IDs:

Edit: I'm now not sure what the best method is. Previously, I suggested lspci -k | grep -A 2 -E "(VGA|3D)". However, this does not give correct IDs on my new Threadripper system.

What worked was to sudo startx, open /var/log/Xorg.0.log (or whatever location startX lists in its output under the line "Log file:"), and look for the line NVIDIA(0): Valid display device(s) on GPU-<GPU number> at PCI:<PCI ID>.

Edit /etc/X11/xorg.conf

Here is an example for a three-GPU machine:

Section "ServerLayout"
        Identifier "dual"
        Screen 0 "Screen0"
        Screen 1 "Screen1" RightOf "Screen0"
        Screen 1 "Screen2" RightOf "Screen1"
EndSection

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BusID          "PCI:5:0:0"
    Option         "Coolbits"       "7"
    Option         "AllowEmptyInitialConfiguration"
EndSection

Section "Device"
    Identifier     "Device1"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BusID          "PCI:6:0:0"
    Option         "Coolbits"       "7"
    Option         "AllowEmptyInitialConfiguration"
EndSection

Section "Device"
    Identifier     "Device2"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BusID          "PCI:9:0:0"
    Option         "Coolbits"       "7"
    Option         "AllowEmptyInitialConfiguration"
EndSection

Section "Screen"
        Identifier     "Screen0"
        Device         "Device0"
EndSection

Section "Screen"
        Identifier     "Screen1"
        Device         "Device1"
EndSection

Section "Screen"
        Identifier     "Screen2"
        Device         "Device2"
EndSection

The BusID must match the PCI ID we identified in the previous step. The option AllowEmptyInitialConfiguration allows X to start even if no monitor is connected. The option Coolbits allows fans to be controlled. It can also allow overclocking, but this has not been tested by me.

Edit /root/.xinitrc

nvidia-settings -q fans
nvidia-settings -a [gpu:0]/GPUFanControlState=1 -a [fan:0]/GPUTargetFanSpeed=75
nvidia-settings -a [gpu:1]/GPUFanControlState=1 -a [fan:1]/GPUTargetFanSpeed=75
nvidia-settings -a [gpu:2]/GPUFanControlState=1 -a [fan:2]/GPUTargetFanSpeed=75

cat

I use .xinitrc to execute nvidia-settings for convenience, although there's probably other ways. Here, I set the fans to 75%. I prevent X server from closing with the empty cat command. This isn't strictly necessary, but I find I sometimes have problems with cards refusing to leave the P8 low-power state if X closes. The first line will print out every GPU fan in the system.

Launch X

sudo startx -- :0

You can execute this command from SSH. The output will be:

Current version of pixman: 0.34.0
    Before reporting problems, check http://wiki.x.org
    to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
    (++) from command line, (!!) notice, (II) informational,
    (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.0.log", Time: Sat May 27 02:22:08 2017
(==) Using config file: "/etc/X11/xorg.conf"
(==) Using system config directory "/usr/share/X11/xorg.conf.d"

  Attribute 'GPUFanControlState' (pushistik:0[gpu:0]) assigned value 1.

  Attribute 'GPUTargetFanSpeed' (pushistik:0[fan:0]) assigned value 75.


  Attribute 'GPUFanControlState' (pushistik:0[gpu:1]) assigned value 1.

  Attribute 'GPUTargetFanSpeed' (pushistik:0[fan:1]) assigned value 75.


  Attribute 'GPUFanControlState' (pushistik:0[gpu:2]) assigned value 1.

  Attribute 'GPUTargetFanSpeed' (pushistik:0[fan:2]) assigned value 75.

Monitor temperatures and clock speeds

nvidia-smi can be used to observe temperatures and power draw. Lower temperatures will allow the card to clock higher and increase its power draw. You can use sudo nvidia-smi -pl 150 to limit power draw and keep the cards cool, or use sudo nvidia-smi -pl 300 to let them overclock. My 1080 Ti runs at 1480 MHz if given 150W, and over 1800 MHz if given 300W, but this depends on the workload. You can monitor their clock speed with nvidia-smi -q or more specifically, watch 'nvidia-smi -q | grep -E "Utilization| Graphics|Power Draw"'

Returning to automatic fan management.

Reboot. I haven't found another way to make the fans automatic.

|improve this answer
  • Are you able to individually control fans on each GPU this way? I've seen that EVGA 1080 Ti cards using this method only change the #1 ( of 3 fans ). Any recommendations for Linux-friendly brands that don't use non-standard Methods of configuration (such as EVGA does)? – ehiller Sep 26 '17 at 15:23
  • @ehiller I only have Founder's Edition cards which have a single blower-style fan. Such a cooler works better when multiple cards are in a system. What does nvidia-settings -q fans show? (You can execute it inside .xinitrc) Maybe you can adjust all the fans if you address them correctly. – Aleksandr Dubinsky Sep 29 '17 at 15:20
  • 1
    @Arin Not sure, but is AllowEmptyInitialConfiguration properly set for the correct PCI ID in xorg.conf? That file can be tricky. – Aleksandr Dubinsky Feb 25 '18 at 20:25
  • 1
    There'a a tool provided by NVIDIA to generate the xorg.conf file automatically and it works pretty great: nvidia-xconfig --allow-empty-initial-configuration --enable-all-gpus --cool-bits=28 --separate-x-screens – Hubert Perron Aug 16 '18 at 13:29
  • 1
    Afaik you can't undervolt NVIDIA GPUs without messing with custom BIOS. Instead i'm setting a software power limit value in watt: nvidia-smi -pl 120. This will limit power consumption of every GPU to 120W. You can also overclock/underclock core and mem using nvidia-smi or nvidia-settings. There's overclocking example in the nvidia-smi doc here: devblogs.nvidia.com/… If you have nvidia-settings installed you can apply a clock offset without having to use specific clock: nvidia-settings -a [gpu]/GPUGraphicsClockOffset[3]=100 – Hubert Perron Aug 24 '18 at 15:56
2

I've written a pip-installable Python script to do something similar to @AlexsandrDubinsky's suggestion.

When you run fans.py, it sets up a temporary X server for each GPU with a fake display attached. Then, it loops over the GPUs every few seconds and sets the fan speed according to their temperature. When the script dies, it returns control of the fans to the drivers and cleans up the X servers.

|improve this answer
  • Thanks! Something which I tried to figure out but couldn't is how to do this whole thing inside Docker and avoid installing X server. Have you tried this? – Aleksandr Dubinsky Oct 28 '19 at 15:36
  • Haven't tried it I'm afraid, and I can't think of a way to do it without X server being accessible. Would have to set the fan speed by another way than nvidia-settings – Andy Jones Oct 29 '19 at 13:05
  • Well, the nvidia driver is accessible within containers and can be used for running eg CUDA. I think it should be possible to launch X server and have it talk to the nvidia driver, but I haven't had luck so far. – Aleksandr Dubinsky Oct 30 '19 at 10:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.