How is it possible to control the fan speed of multiple consumer NVIDIA GPUs such as Titan and 1080 Ti on a headless node running Linux?
3 Answers
The following is a simple method that does not require scripting, connecting fake monitors, or fiddling and can be executed over SSH to control multiple NVIDIA GPUs' fans. It has been tested on Arch Linux.
Create xorg.conf
sudo nvidia-xconfig --allow-empty-initial-configuration --enable-all-gpus --cool-bits=7
This will create an /etc/X11/xorg.conf
with an entry for each GPU, similar to the manual method.
Note: Some distributions (Fedora, CentOS, Manjaro) have additional config files (eg in /etc/X11/xorg.conf.d/
or /usr/share/X11/xorg.conf.d/
), which override xorg.conf
and set AllowNVIDIAGPUScreens
. This option is not compatible with this guide. The extra config files should be modified or deleted. The X11 log file shows which config files have been loaded.
Alternative: Create xorg.conf manually
Identify your cards' PCI IDs:
nvidia-xconfig --query-gpu-info
Find the PCI BusID
fields. Note that these are not the same as the bus IDs reported in the kernel.
Alternatively, do sudo startx
, open /var/log/Xorg.0.log
(or whatever location startX lists in its output under the line "Log file:"), and look for the line NVIDIA(0): Valid display device(s) on GPU-<GPU number> at PCI:<PCI ID>
.
Edit /etc/X11/xorg.conf
Here is an example of xorg.conf
for a three-GPU machine:
Section "ServerLayout"
Identifier "dual"
Screen 0 "Screen0"
Screen 1 "Screen1" RightOf "Screen0"
Screen 1 "Screen2" RightOf "Screen1"
EndSection
Section "Device"
Identifier "Device0"
Driver "nvidia"
VendorName "NVIDIA Corporation"
BusID "PCI:5:0:0"
Option "Coolbits" "7"
Option "AllowEmptyInitialConfiguration"
EndSection
Section "Device"
Identifier "Device1"
Driver "nvidia"
VendorName "NVIDIA Corporation"
BusID "PCI:6:0:0"
Option "Coolbits" "7"
Option "AllowEmptyInitialConfiguration"
EndSection
Section "Device"
Identifier "Device2"
Driver "nvidia"
VendorName "NVIDIA Corporation"
BusID "PCI:9:0:0"
Option "Coolbits" "7"
Option "AllowEmptyInitialConfiguration"
EndSection
Section "Screen"
Identifier "Screen0"
Device "Device0"
EndSection
Section "Screen"
Identifier "Screen1"
Device "Device1"
EndSection
Section "Screen"
Identifier "Screen2"
Device "Device2"
EndSection
The BusID
must match the bus IDs we identified in the previous step. The option AllowEmptyInitialConfiguration
allows X to start even if no monitor is connected. The option Coolbits
allows fans to be controlled. It can also allow overclocking.
Note: Some distributions (Fedora, CentOS, Manjaro) have additional config files (eg in /etc/X11/xorg.conf.d/
or /usr/share/X11/xorg.conf.d/
), which override xorg.conf
and set AllowNVIDIAGPUScreens
. This option is not compatible with this guide. The extra config files should be modified or deleted. The X11 log file shows which config files have been loaded.
Edit /root/.xinitrc
nvidia-settings -q fans
nvidia-settings -a [gpu:0]/GPUFanControlState=1 -a [fan:0]/GPUTargetFanSpeed=75
nvidia-settings -a [gpu:1]/GPUFanControlState=1 -a [fan:1]/GPUTargetFanSpeed=75
nvidia-settings -a [gpu:2]/GPUFanControlState=1 -a [fan:2]/GPUTargetFanSpeed=75
I use .xinitrc to execute nvidia-settings for convenience, although there's probably other ways. The first line will print out every GPU fan in the system. Here, I set the fans to 75%.
Launch X
sudo startx -- :0
You can execute this command from SSH. The output will be:
Current version of pixman: 0.34.0
Before reporting problems, check http://wiki.x.org
to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
(++) from command line, (!!) notice, (II) informational,
(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.0.log", Time: Sat May 27 02:22:08 2017
(==) Using config file: "/etc/X11/xorg.conf"
(==) Using system config directory "/usr/share/X11/xorg.conf.d"
Attribute 'GPUFanControlState' (pushistik:0[gpu:0]) assigned value 1.
Attribute 'GPUTargetFanSpeed' (pushistik:0[fan:0]) assigned value 75.
Attribute 'GPUFanControlState' (pushistik:0[gpu:1]) assigned value 1.
Attribute 'GPUTargetFanSpeed' (pushistik:0[fan:1]) assigned value 75.
Attribute 'GPUFanControlState' (pushistik:0[gpu:2]) assigned value 1.
Attribute 'GPUTargetFanSpeed' (pushistik:0[fan:2]) assigned value 75.
Monitor temperatures and clock speeds
nvidia-smi
and nvtop
can be used to observe temperatures and power draw. Lower temperatures will allow the card to clock higher and increase its power draw. You can use sudo nvidia-smi -pl 150
to limit power draw and keep the cards cool, or use sudo nvidia-smi -pl 300
to let them overclock. My 1080 Ti runs at 1480 MHz if given 150W, and over 1800 MHz if given 300W, but this depends on the workload. You can monitor their clock speed with nvidia-smi -q
or more specifically, watch 'nvidia-smi -q | grep -E "Utilization| Graphics|Power Draw"'
Returning to automatic fan management.
Reboot. I haven't found another way to make the fans automatic.
-
Are you able to individually control fans on each GPU this way? I've seen that EVGA 1080 Ti cards using this method only change the #1 ( of 3 fans ). Any recommendations for Linux-friendly brands that don't use non-standard Methods of configuration (such as EVGA does)?– ehillerSep 26, 2017 at 15:23
-
@ehiller I only have Founder's Edition cards which have a single blower-style fan. Such a cooler works better when multiple cards are in a system. What does
nvidia-settings -q fans
show? (You can execute it inside .xinitrc) Maybe you can adjust all the fans if you address them correctly. Sep 29, 2017 at 15:20 -
1@Arin Not sure, but is
AllowEmptyInitialConfiguration
properly set for the correct PCI ID in xorg.conf? That file can be tricky. Feb 25, 2018 at 20:25 -
1There'a a tool provided by NVIDIA to generate the xorg.conf file automatically and it works pretty great:
nvidia-xconfig --allow-empty-initial-configuration --enable-all-gpus --cool-bits=28 --separate-x-screens
Aug 16, 2018 at 13:29 -
1Afaik you can't undervolt NVIDIA GPUs without messing with custom BIOS. Instead i'm setting a software power limit value in watt:
nvidia-smi -pl 120
. This will limit power consumption of every GPU to 120W. You can also overclock/underclock core and mem usingnvidia-smi
ornvidia-settings
. There's overclocking example in thenvidia-smi
doc here: devblogs.nvidia.com/… If you havenvidia-settings
installed you can apply a clock offset without having to use specific clock:nvidia-settings -a [gpu]/GPUGraphicsClockOffset[3]=100
Aug 24, 2018 at 15:56
When you run fans.py, it sets up a temporary X server for each GPU with a fake display attached. Then, it loops over the GPUs every few seconds and sets the fan speed according to their temperature. When the script dies, it returns control of the fans to the drivers and cleans up the X servers.
-
Thanks! Something which I tried to figure out but couldn't is how to do this whole thing inside Docker and avoid installing X server. Have you tried this? Oct 28, 2019 at 15:36
-
Haven't tried it I'm afraid, and I can't think of a way to do it without X server being accessible. Would have to set the fan speed by another way than
nvidia-settings
Oct 29, 2019 at 13:05 -
Well, the nvidia driver is accessible within containers and can be used for running eg CUDA. I think it should be possible to launch X server and have it talk to the nvidia driver, but I haven't had luck so far. Oct 30, 2019 at 10:30
-
Have you by any chance heard about any new features for this? With the current heat I can only use the two outer GPUs of my 4 Titan XP setup for CUDA computations, and even then they clock down due to high heat. I was hoping that I could manually control the inner two GPU's fan speed to help cooling the outer two, but it seems a hassle to set up? Edit: Also working in a docker instance on our compute cluster. Aug 13, 2020 at 11:40
-
If I'm understanding you right then that's too specialised to be something I'd add to coolgpus itself, but it's a simple script and you should be able to fork it and adapt it to your purposes. Open an issue if you get stuck. NB: I'd be surprised if you could adjust the fan settings from within a Docker container though, usually you need sudo. Aug 13, 2020 at 14:58
Based on answers to this question and similar StackExchange questions, I wrote a shell script that will set the fan speed to 100
(or whatever value you want) to all of your fans on all of your GPUs on a machine.
This script assumes your machine has X11 installed, but that you aren't using it to serve a GUI to a user.
/bin/set-gpu-fan-speed.sh:
#!/bin/bash
set -Eeuxo pipefail
# Kill any existing X servers.
killall Xorg || true
sleep 5
# Create a NVIDIA-friendly Xorg config.
nvidia-xconfig -a --cool-bits=28 --allow-empty-initial-configuration --enable-all-gpus
# Start a new X server for nvidia-settings to use.
export XDG_SESSION_TYPE=x11
export DISPLAY=:0
startx -- $DISPLAY &
sleep 5
# Determine the number of GPUs and fans on this machine.
NUM_GPUS=$(nvidia-settings -q gpus | grep -c 'gpu:')
NUM_FANS=$(nvidia-settings -q fans | grep -c 'fan:')
# For each GPU, enable fan control.
for ((i=0; i < NUM_GPUS; i++))
do
nvidia-settings --verbose=all -a "[gpu:$i]/GPUFanControlState=1"
done
# For each fan, set fan speed to 100%.
for ((i=0; i < NUM_FANS; i++))
do
nvidia-settings --verbose=all -a "[fan:$i]/GPUTargetFanSpeed=100"
done
# Kill the X server that we started.
killall Xorg || true
These fan speed changes do not persist across reboots, so I've written a systemd unit file to run the above script on every boot.
/etc/systemd/system/set-gpu-fan-speed.service:
[Unit]
Description="Sets the GPU fan speed"
[Service]
Type=oneshot
User=root
ExecStart=/bin/set-gpu-fan-speed.sh
[Install]
WantedBy=multi-user.target
After creating the above file, run the following commands as root to enable the script running on reboot.
systemctl enable set-gpu-fan-speed.service
systemctl start set-gpu-fan-speed.service
-
-
@techcheese: I am not sure. But if you're using your laptop's graphical display like a normal person, I guess you could skip the parts of my script that involve killing or starting Xorg. Apr 3, 2021 at 5:18