Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am running my Python script in the background in my Ubuntu machine (12.04) like this -

nohup python testing.py > test.out &

Now, it might be possible that at some stage my above Python script can die for whatever reason.

So I am thinking to have some sort of cron agent in bash shell script which can restart my above Python script automatically if it is killed for whatever reason.

Is this possible to do? If yes, then what's the best way to solve these kind of problem?

UPDATE:

After creating the testing.conf file like this -

chdir /tekooz
exec python testing.py
respawn

I ran below sudo command to start it but I cannot see that process running behind using ps ax?

root@bx13:/bezook# sudo start testing
testing start/running, process 27794
root@bx13:/bezook# ps ax | grep testing.py
27806 pts/3    S+     0:00 grep --color=auto testing.py

Any idea why px ax is not showing me anything? And how do I check whether my program is running or not?

This is my python script -

#!/usr/bin/python
while True:
    print "Hello World"
    time.sleep(5)
share|improve this question

7 Answers 7

up vote 7 down vote accepted

On Ubuntu can use upstart to do so, better than a cron job. You put a config setup in /etc/init and make sure you specify respawn

It could be a minimal file /etc/init/testing.conf (edit as root):

chdir /your/base/directory
exec python testing.py
respawn

And you can test with /your/base/directory/testing.py:

from __future__ import print_function

import time

with open('/var/tmp/testing.log', 'a') as fp:
    print(time.time(), 'done', file=fp)
    time.sleep(3)

and start with:

sudo start testing

and follow what happens (in another window) with:

tail -f /var/tmp/testing.log

and stop with:

sudo stop testing

You can also add [start on][2] to have the command start on boot of the system.

share|improve this answer
    
If you use a cron job then you'll want to either implement or find some code for robust PID file handling. You want to have your service/script/daemon create a PID file (conventionally located under /var/run) and have its start-up code check if the file contents are stale (left by a killed process). This sort of code is surprisingly difficult to write free of races and corner cases. stackoverflow.com/questions/788411/… –  Jim Dennis Jan 5 at 8:08
    
@Zelda: Thanks for suggestion.. I am new to Linux/Unix world.. What type of changes I am supposed to make in /etc/init file? If you can provide step by step guide for me, then I will be able to learn something and do the right thing.. –  Webby Jan 5 at 8:24
    
@Webby I made the answer more complete. If you do not want to open a file for output and rewrite your print statements you can do something like sys.stdout = open(file_name, 'w') at the beginning. –  Zelda Jan 5 at 10:29
    
Thank You Zelda. Appreciated your help.. I updated the question with some details.. I am trying to do like this to see whether my testing.py is running or not.. It doesn't shows me whether it is running or not.. px ax | grep testing.py.. It is returning me nothing? Any idea why? –  Webby Jan 5 at 18:58
    
You should put the whole thing in a try/except clause and write to a logfile what exception was generated and that the program exits. Maybe the print statement does not work as it cannot write to stdout. –  Zelda Jan 5 at 19:35

You shouldn't really use this for production, but you could:

#!/bin/sh

while true; do
  nohup python testing.py >> test.out
done &

If, for any reason, python process exits, the shell loop will continue and restart it, appending to the .out file as desired. Nearly no overhead and takes very little time to set up.

share|improve this answer

You need a supervisor, you can use supervisor. It is python based supervisor, therefore easy to modify if you need to.

Control is with files with .ini file syntax.

share|improve this answer

You can also use monit Or Process monitoring with ps-watcher

Monit is an open source utility for managing and monitoring, processes, programs, files, directories and filesystems on a UNIX system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations.

Here is example for your scenario:

check process myprocessname
        matching "myprocessname"
        start program = "nohup /usr/bin/python /path/testing.py > /tmp/test.out &"
        stop program = "/usr/bin/killall myprocessname"

Take look at monit examples

share|improve this answer

You could also take a more shell oriented approach. Have your cron look for your script and relaunch it if it dies.

  1. Create a new crontab by running crontab -e. This will bring up a window of your favorite text editor.

  2. Add this line to the file that just opened

    */5 * * * * pgrep -f testing.py || nohup python /home/you/scripts/testing.py > test.out
    
  3. Save the file and exit the editor.

You just created a new crontab which will be run every 5 minutes and launch your script unless it is already running. See here for a nice little tutorial on cron. The official Ubuntu docs on cron are here.

The actual command being run is pgrep which searches running processes for the string given in the command line. pgrep foo will search for a program named foo and return it's process identifier. pgrep -f makes it search the entire command line used to launch the program and not only the program name (useful because this is a python script).

The || symbol means "do this if the previous command failed". So, if your script is not running, the pgrep will fail since it will find nothing and your script will be launched.

share|improve this answer
    
Thank you.. But I am new to linux and unix so don't know where is crontab? Is this a file in my ubuntu machine somewhere? –  Webby Jan 5 at 9:26
    
@Webby see updated answer. –  terdon Jan 5 at 9:31
    
Thanks terdon.. I can run this command crontab -e from the directory where my python script is.. Correct? –  Webby Jan 5 at 9:42
    
@Webby you can run it from anywhere you want. cron is a scheduling daemon, it is a service that runs in the background. If your python script is not in your $PATH (if you can't launch it from anywhere but need to be in it's directory) use the full path to the script as in my updated answer. –  terdon Jan 5 at 9:44
    
Thanks. Now it makes sense.. I just created a new crontab and edited the file by adding the same single line but for 1 minute.. I have already created a Hello World Python script revolving around while True named as testing.py.. After saving the crontab file, it should automatically start the testing.py after 1 minute? And then keep on checking every 1 minute whether python script is running or not? If yes, after saving the crontab -e file, I did ps ax | grep testing.py and I am not able to see any process for that? –  Webby Jan 5 at 10:11

There are a number of ways to monitor and respawn processes under UNIX/Linux. One of the oldest is a "respawn" entry in /etc/inittab ... if you're using the old SysV init system. Another, means is to use the supervisor daemon from DJ Bernstein's daemontools package. Other options are to use features in Ubuntu upstart ... or systemd or others.

But you can look at alternatives init and in the Python code for Pardus: mudur daemon in particular.

If you decide to go with a cron job (and PID file handling) then consider reading this PEP 3143 and perhaps using its reference implementation.

As I alluded to in my other comments, robust PID file handling is tricky. It's prone to races and corner cases. It get's trickier if there's any chance that your PID file ends up on and NFS or other networked filesystem (some of the atomicity guarantees you get with the file handling semantics on proper local UNIX/Linux filesystems go away on some versions and implementations of NFS, for example). Also the semantics around file locking under UNIX can be tricky. (Does an flock or fcntl lock get released promptly, in your target OS, when the process holding it is killed with SIGKILL, for example?).

share|improve this answer

You can have the testing program redirect the output using a commandline option and then use a simple python script to restart the program indefinitely:

import subprocess

while True:
    try:
        print subprocess.check_output(['python', 'testing.py'])
    except KeyboardInterrupt:
        break

you can put this program in the background, and once you want to stop just pull it into the foreground and kill it.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.