How to restart the Python script automatically if it is killed or dies

Question

I am running my Python script in the background in my Ubuntu machine (12.04) like this -

nohup python testing.py > test.out &

Now, it might be possible that at some stage my above Python script can die for whatever reason.

So I am thinking to have some sort of cron agent in bash shell script which can restart my above Python script automatically if it is killed for whatever reason.

Is this possible to do? If yes, then what's the best way to solve these kind of problem?

UPDATE:

After creating the testing.conf file like this -

chdir /tekooz
exec python testing.py
respawn

I ran below sudo command to start it but I cannot see that process running behind using ps ax?

root@bx13:/bezook# sudo start testing
testing start/running, process 27794
root@bx13:/bezook# ps ax | grep testing.py
27806 pts/3    S+     0:00 grep --color=auto testing.py

Any idea why px ax is not showing me anything? And how do I check whether my program is running or not?

This is my python script -

#!/usr/bin/python
while True:
    print "Hello World"
    time.sleep(5)

Answer 1

On Ubuntu (until 14.04, 16.04 and later use systemd) can use upstart to do so, better than a cron job. You put a config setup in /etc/init and make sure you specify respawn

It could be a minimal file /etc/init/testing.conf (edit as root):

chdir /your/base/directory
exec python testing.py
respawn

And you can test with /your/base/directory/testing.py:

from __future__ import print_function

import time

with open('/var/tmp/testing.log', 'a') as fp:
    print(time.time(), 'done', file=fp)
    time.sleep(3)

and start with:

sudo start testing

and follow what happens (in another window) with:

tail -f /var/tmp/testing.log

and stop with:

sudo stop testing

You can also add [start on][2] to have the command start on boot of the system.

Answer 2

You could also take a more shell oriented approach. Have your cron look for your script and relaunch it if it dies.

1. Create a new crontab by running crontab -e. This will bring up a window of your favorite text editor.

2. Add this line to the file that just opened

   */5 * * * * pgrep -f testing.py || nohup python /home/you/scripts/testing.py > test.out
   

3. Save the file and exit the editor.

You just created a new crontab which will be run every 5 minutes and launch your script unless it is already running. See here for a nice little tutorial on cron. The official Ubuntu docs on cron are here.

The actual command being run is pgrep which searches running processes for the string given in the command line. pgrep foo will search for a program named foo and return its process identifier. pgrep -f makes it search the entire command line used to launch the program and not only the program name (useful because this is a python script).

The || symbol means "do this if the previous command failed". So, if your script is not running, the pgrep will fail since it will find nothing and your script will be launched.

Answer 3

You shouldn't really use this for production, but you could:

#!/bin/sh

while true; do
  nohup python testing.py >> test.out
done &

If, for any reason, python process exits, the shell loop will continue and restart it, appending to the .out file as desired. Nearly no overhead and takes very little time to set up.

Answer 4

You can have the testing program redirect the output using a commandline option and then use a simple python script to restart the program indefinitely:

import subprocess

while True:
    try:
        print subprocess.check_output(['python', 'testing.py'])
    except KeyboardInterrupt:
        break

you can put this program in the background, and once you want to stop just pull it into the foreground and kill it.

Answer 5

There are a number of ways to monitor and respawn processes under UNIX/Linux. One of the oldest is a "respawn" entry in /etc/inittab ... if you're using the old SysV init system. Another method is to use the supervisor daemon from DJ Bernstein's daemontools package. Other options are to use features in Ubuntu upstart ... or systemd or others.

But you can look at alternatives init and in the Python code for Pardus: mudur daemon in particular.

If you decide to go with a cron job (and PID file handling) then consider reading this PEP 3143 and perhaps using its reference implementation.

As I alluded to in my other comments, robust PID file handling is tricky. It's prone to races and corner cases. It gets trickier if there's any chance that your PID file ends up on an NFS or other networked filesystem (some of the atomicity guarantees you get with the file handling semantics on proper local UNIX/Linux filesystems go away on some versions and implementations of NFS, for example). Also the semantics around file locking under UNIX can be tricky. (Does an flock or fcntl lock get released promptly, in your target OS, when the process holding it is killed with SIGKILL, for example?).

Answer 6

You can also use monit Or Process monitoring with ps-watcher

Monit is an open source utility for managing and monitoring, processes, programs, files, directories and filesystems on a UNIX system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations.

Here is example for your scenario:

check process myprocessname
        matching "myprocessname"
        start program = "nohup /usr/bin/python /path/testing.py > /tmp/test.out &"
        stop program = "/usr/bin/killall myprocessname"

Take look at monit examples

Answer 7

You need a supervisor, you can use supervisor. It is python based supervisor, therefore easy to modify if you need to.

Control is with files with .ini file syntax.

Answer 8

Terdon's answer, did not work for me, because pgrep -f testing.py was never 'failing'. It would grab the pid for the cron job (because of the -f option). However, without the -f option pgrep won't find testing.py because there's no process called testing.py.

My solution to this was to change

pgrep -f testing.py

to

pgrep -f testing.py | pgrep python

this means the full crontab job would be:

*/5 * * * * pgrep -f testing.py | pgrep python || nohup python /home/you/scripts/testing.py > test.out

Answer 9

In my case, as a quick-fix, I wanted to keep my program running when it exited with en error or it was killed. On the other hand, I wanted to stop the execution when the program terminated correctly (return code = 0)

I have tested it on Bash. It should work fine in any other shell

#!/bin/sh

echo ""
echo "Use: $0 ./instagram.py"
echo ""

echo "Executing $1 ..."

EXIT_CODE=1
(while [ $EXIT_CODE -gt 0 ]; do
    $1
    # loops on error code: greater-than 0
    EXIT_CODE=$?
done)

Answer 10

For terdon's answer, pgrep -f testing.py will never return false according to the comments in here:

I think the issue is that cron spawns a shell to run your command, and the arguments of that shell are matched by pgrep since you are using -f

For Matt's answer, pgrep -f testing.py is useless since pgrep python matches any running Python script. So if two Python script cronjob, the second cronjob will never run.

And then I found the solution to solve pgrep -f testing.py in the comment here: https://askubuntu.com/questions/1014559/running-pgrep-in-a-crontab?noredirect=1&lq=1

My cron for running two Python scripts:

* * * * * pgrep -f '^/usr/bin/python36 /home/ec2-user/myscript1\.py' || nohup /usr/bin/python36 /home/ec2-user/myscript1.py

0 * * * * pgrep -f '^/usr/bin/python36 /home/ec2-user/myscript2\.py' || nohup /usr/bin/python36 /home/ec2-user/myscript2.py

Answer 11

In Ubuntu this works for me thanks to --wait

#!/bin/bash

while :
do
  sleep 5
  gnome-terminal --wait -- sh -c "python3 myscript.py 'myarg1'"
done

Answer 12

There's a Python module for that, forever.

The advantage being, hopefully, in using the same language for both the code and the watchdog. If it needs to be improved, one can find it in

cd $(python -c "import site; print(site.getusersitepackages())")

I'd install it with

python -mpip install --user --upgrade forever

and later use it with

python -mforever.run -t 9 -i 9 python script-to-watch.py