1

I am running multiple GPUs (GTX 1060 x6) and two are not responding. I queried the drivers with lspci -nnk (output below), which showed that 4 of the GPUs had "Kernel driver in use: nvidia" while the other two had no listing of "Kernel driver". I'm running Ubuntu 16.04 LTS on 4.4.0-104-generic with CUDA 8.0 installed for tensor flow and nvidia-387 driver (open source). Any ideas why the kernel driver is not showing up?

00:00.0 Host bridge [0600]: Intel Corporation Sky Lake Host Bridge/DRAM Registers [8086:190f] (rev 07)
    Subsystem: ASUSTeK Computer Inc. Skylake Host Bridge/DRAM Registers [1043:8694]
00:01.0 PCI bridge [0604]: Intel Corporation Sky Lake PCIe Controller (x16) [8086:1901] (rev 07)
    Kernel driver in use: pcieport
    Kernel modules: shpchp
00:14.0 USB controller [0c03]: Intel Corporation Device [8086:a2af]
    Subsystem: ASUSTeK Computer Inc. Device [1043:8694]
    Kernel driver in use: xhci_hcd
00:16.0 Communication controller [0780]: Intel Corporation Device [8086:a2ba]
    Subsystem: ASUSTeK Computer Inc. Device [1043:8694]
    Kernel driver in use: mei_me
    Kernel modules: mei_me
00:17.0 SATA controller [0106]: Intel Corporation Device [8086:a282]
    Subsystem: ASUSTeK Computer Inc. Device [1043:8694]
    Kernel driver in use: ahci
    Kernel modules: ahci
00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:a294] (rev f0)
    Kernel driver in use: pcieport
    Kernel modules: shpchp
00:1c.6 PCI bridge [0604]: Intel Corporation Device [8086:a296] (rev f0)
    Kernel driver in use: pcieport
    Kernel modules: shpchp
00:1c.7 PCI bridge [0604]: Intel Corporation Device [8086:a297] (rev f0)
    Kernel driver in use: pcieport
    Kernel modules: shpchp
00:1d.0 PCI bridge [0604]: Intel Corporation Device [8086:a298] (rev f0)
    Kernel driver in use: pcieport
    Kernel modules: shpchp
00:1d.2 PCI bridge [0604]: Intel Corporation Device [8086:a29a] (rev f0)
    Kernel driver in use: pcieport
    Kernel modules: shpchp
00:1d.3 PCI bridge [0604]: Intel Corporation Device [8086:a29b] (rev f0)
    Kernel driver in use: pcieport
    Kernel modules: shpchp
00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a2c8]
    Subsystem: ASUSTeK Computer Inc. Device [1043:8694]
00:1f.2 Memory controller [0580]: Intel Corporation Device [8086:a2a1]
    Subsystem: ASUSTeK Computer Inc. Device [1043:8694]
00:1f.3 Audio device [0403]: Intel Corporation Device [8086:a2f0]
    Subsystem: ASUSTeK Computer Inc. Device [1043:8723]
    Kernel driver in use: snd_hda_intel
    Kernel modules: snd_hda_intel
00:1f.4 SMBus [0c05]: Intel Corporation Device [8086:a2a3]
    Subsystem: ASUSTeK Computer Inc. Device [1043:8694]
    Kernel modules: i2c_i801
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I219-V [8086:15b8]
    Subsystem: ASUSTeK Computer Inc. Ethernet Connection (2) I219-V [1043:8672]
    Kernel driver in use: e1000e
    Kernel modules: e1000e
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1c03] (rev a1)
    Subsystem: eVga.com. Corp. Device [3842:6163]
    Kernel driver in use: nvidia
    Kernel modules: nvidiafb, nouveau, nvidia_387_drm, nvidia_387
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f1] (rev a1)
    Subsystem: eVga.com. Corp. Device [3842:6163]
    Kernel driver in use: snd_hda_intel
    Kernel modules: snd_hda_intel
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1c03] (rev a1)
    Subsystem: eVga.com. Corp. Device [3842:6161]
    Kernel driver in use: nvidia
    Kernel modules: nvidiafb, nouveau, nvidia_387_drm, nvidia_387
02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f1] (rev a1)
    Subsystem: eVga.com. Corp. Device [3842:6161]
    Kernel driver in use: snd_hda_intel
    Kernel modules: snd_hda_intel
03:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1c03] (rev a1)
    Subsystem: eVga.com. Corp. Device [3842:6163]
    Kernel driver in use: nvidia
    Kernel modules: nvidiafb, nouveau, nvidia_387_drm, nvidia_387
03:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f1] (rev a1)
    Subsystem: eVga.com. Corp. Device [3842:6163]
    Kernel driver in use: snd_hda_intel
    Kernel modules: snd_hda_intel
04:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1c03] (rev a1)
    Subsystem: eVga.com. Corp. Device [3842:6161]
    Kernel driver in use: nvidia
    Kernel modules: nvidiafb, nouveau, nvidia_387_drm, nvidia_387
04:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f1] (rev a1)
    Subsystem: eVga.com. Corp. Device [3842:6161]
    Kernel driver in use: snd_hda_intel
    Kernel modules: snd_hda_intel
06:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1c03] (rev a1)
    Subsystem: eVga.com. Corp. Device [3842:6163]
    Kernel modules: nvidiafb, nouveau, nvidia_387_drm, nvidia_387
06:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f1] (rev a1)
    Subsystem: eVga.com. Corp. Device [3842:6163]
    Kernel driver in use: snd_hda_intel
    Kernel modules: snd_hda_intel
07:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1c03] (rev a1)
    Subsystem: eVga.com. Corp. Device [3842:6163]
    Kernel modules: nvidiafb, nouveau, nvidia_387_drm, nvidia_387
07:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f1] (rev a1)
    Subsystem: eVga.com. Corp. Device [3842:6163]
    Kernel driver in use: snd_hda_intel
    Kernel modules: snd_hda_intel
1

Update: The solution was messing with the BIOS to turn on all the PCIe lanes for use as well as using Above 4G Decoding Enabled

0

You need to get rid of the 'nouveau' module/driver.

Assuming you are using gnome, open "Software & Updates" -> "Additional Drivers" and change it to one of the "NVIDIA binary drivers"

If that doesn't work for you, the only option is to go through the blacklisting method documented here.

http://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.