2

I'm running a headless Ubuntu server 14.04 with GeForce 690 GPU, trying to do fun Cuda machine learning stuff. I'm having issues with the driver hanging, and hoping to get some help debugging it here. This is a new issue; the card has functioned correctly in the past for computing.

The problem is that basically any program that tries to connect to the card hangs. It happens with very simple programs I've written myself, as well as with nvidia's provided nvidia-smi binary. I've run each of these through strace and they all hang on an open syscall, attempting to open the /dev/nvidiactl device.

Once the programs are hung, they are no longer able to receive signals (kill -9 has no effect). I can only stop them by rebooting.

I've tried to force rmmod of the nvidia driver, but it claims to be in use. lsof shows no programs actually holding a file descriptor for /dev/nvidiactl.

There is no display attached to the machine, and no graphical programs (X, gnome, unity, etc) are running.

What's the best way to even begin debugging something like this at a system level? It's tough since the driver code is proprietary...

5
  • I'm facing a similar situation. Usually killall -s 9 works after 1-2 minutes. Unfortunately I'm working on a cluster so I don't have root access. Commented Mar 2, 2016 at 16:54
  • This is happening to me, too! Trying to open any of the /dev/nvidia* devices leads to what I've been calling "basilisk syndrome". strace shows that it makes the syscall, then hangs forever waiting for a response. Reboots fix it, but it seems to return at random. Commented May 11, 2016 at 4:31
  • I have a very vague suspicion that this problem goes away when I connect the GPU to a display. I say it's a vague suspicion because I can't reliably reproduce the failure in either state (connected or not connected to display). After giving up on this for a long time, I recently started hacking on the machine again, hoping to solve this problem. I was pleased/annoyed to find the problem had magically disappeared. Everything's been working fine for a few weeks, then last night it suddenly happened again. This morning it was happening reliably; I connected to a display and it's working for now. Commented Sep 3, 2016 at 20:56
  • Ok I've debunked that theory. The GPU ran fine just now, for about an hour, training a tensorflow model. I ctrl-c'ed it, and at that point the card is hung. Still connected to a display. My next suspect is the motherboard. I was already planning to replace it, so I'll do that in the coming week and update here if the issue goes away. Commented Sep 3, 2016 at 21:14
  • Ah! Another clue. dmesg shows some evidence of a multiple pointer dereference in the nvidia driver pastebin.com/gJFR2Dkq Commented Sep 3, 2016 at 21:23

0

You must log in to answer this question.

Browse other questions tagged .