2

I'm running a headless Ubuntu server 14.04 with GeForce 690 GPU, trying to do fun Cuda machine learning stuff. I'm having issues with the driver hanging, and hoping to get some help debugging it here. This is a new issue; the card has functioned correctly in the past for computing.

The problem is that basically any program that tries to connect to the card hangs. It happens with very simple programs I've written myself, as well as with nvidia's provided nvidia-smi binary. I've run each of these through strace and they all hang on an open syscall, attempting to open the /dev/nvidiactl device.

Once the programs are hung, they are no longer able to receive signals (kill -9 has no effect). I can only stop them by rebooting.

I've tried to force rmmod of the nvidia driver, but it claims to be in use. lsof shows no programs actually holding a file descriptor for /dev/nvidiactl.

There is no display attached to the machine, and no graphical programs (X, gnome, unity, etc) are running.

What's the best way to even begin debugging something like this at a system level? It's tough since the driver code is proprietary...

  • I'm facing a similar situation. Usually killall -s 9 works after 1-2 minutes. Unfortunately I'm working on a cluster so I don't have root access. – Lorenzo Pistone Mar 2 '16 at 16:54
  • This is happening to me, too! Trying to open any of the /dev/nvidia* devices leads to what I've been calling "basilisk syndrome". strace shows that it makes the syscall, then hangs forever waiting for a response. Reboots fix it, but it seems to return at random. – etherealflux May 11 '16 at 4:31
  • I have a very vague suspicion that this problem goes away when I connect the GPU to a display. I say it's a vague suspicion because I can't reliably reproduce the failure in either state (connected or not connected to display). After giving up on this for a long time, I recently started hacking on the machine again, hoping to solve this problem. I was pleased/annoyed to find the problem had magically disappeared. Everything's been working fine for a few weeks, then last night it suddenly happened again. This morning it was happening reliably; I connected to a display and it's working for now. – Chris Suter Sep 3 '16 at 20:56
  • Ok I've debunked that theory. The GPU ran fine just now, for about an hour, training a tensorflow model. I ctrl-c'ed it, and at that point the card is hung. Still connected to a display. My next suspect is the motherboard. I was already planning to replace it, so I'll do that in the coming week and update here if the issue goes away. – Chris Suter Sep 3 '16 at 21:14
  • Ah! Another clue. dmesg shows some evidence of a multiple pointer dereference in the nvidia driver pastebin.com/gJFR2Dkq – Chris Suter Sep 3 '16 at 21:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.