Keep running program requiring X server after logging off of ssh session

Question

I have a python script that periodically generates a png image, and puts it on a server that I can access through a browser. The image displays the progress of a job I am running on the server. I'm using matplotlib to generate the images.

The python script goes to sleep after generating each image. After an hour, it wakes up, generates a new image, and then goes back to sleep.

I start the script over an ssh session. When I log off, the script gets killed. I've read the other threads covering this problem, but the proposed solutions do not work. The script is killed after I log off, even if I run it with

nohup script.py &> job-monitor.log &

or if I run it like this

script.py
[Ctrl-Z]
disown -h %1
bg 1

The strange thing is, when I log off and log back in, the script is no longer running, and job-monitor.log is empty.

What could be happening, and how do I get the script to continue running after I log off?

Additional information: I think the problem has to do with the plotting library I'm using, matplotlib. I think it may require an X session to produce a png, even though the graphic is just being saved to disk.

Answer 1

If the problem is with matplotlib (that is, your script never provides an answer if you stay connected, or it works because ssh forwards your Xwindow connection), you have to put in your matplotlibrc file:

backend : AGG

This way, the script doesn't need Xwindow to work.

Answer 2

Instead of IO/redirecting use tmux or screen command for session monitoring.

Answer 3

May I humbly suggest that you throw away the time.sleep() part from your script and use either cron or at for periodic and/or delayed invocations?

This simplifies your code, safely removes dependencies on ssh session detachment, and separates the concerns of image generation and scheduling.

Regarding the script dying: I suspect that the script after nohup might not have write access to the log file, either because it lacks write access to the directory where you start it, or because an existing log file is owned by e.g. root and thus inaccessible.

I don't know if disowning a job and then trying to call bg against it should work. I always do the opposite: foo & disown %1; it works exactly in circumstances like yours.

Answer 4

Try redirecting STDIN as well. It's possible the script is reading from stdin, so when you close the shell, stdin goes away, and it dies.

script.py </dev/null &>/dev/null &
disown