I'm running a headless Ubuntu server 14.04 with GeForce 690 GPU, trying to do fun Cuda machine learning stuff. I'm having issues with the driver hanging, and hoping to get some help debugging it here. This is a new issue; the card has functioned correctly in the past for computing.
The problem is that basically any program that tries to connect to the card hangs. It happens with very simple programs I've written myself, as well as with nvidia's provided nvidia-smi binary. I've run each of these through strace and they all hang on an open syscall, attempting to open the /dev/nvidiactl device.
Once the programs are hung, they are no longer able to receive signals (kill -9 has no effect). I can only stop them by rebooting.
I've tried to force rmmod of the nvidia driver, but it claims to be in use. lsof shows no programs actually holding a file descriptor for /dev/nvidiactl.
There is no display attached to the machine, and no graphical programs (X, gnome, unity, etc) are running.
What's the best way to even begin debugging something like this at a system level? It's tough since the driver code is proprietary...