How to restart python script after one second of going down using UPSTART?

Question

I am running my Python script using upstart feature of Ubuntu so that if for whatever reason my Python script dies or gets killed, it can be restarted automatically and everything is working fine -

So I decided to use UPSTART feature of Ubuntu to restart the Python script automatically.

After creating the testing.conf file like this in /etc/init/testing.conf -

start on runlevel [2345]
stop on runlevel [016]

chdir /tekooz
exec python testing.py
respawn

I ran below sudo command to start it and I can see that process running using ps ax and my python script is also running fine.

root@bx13:/tekooz# sudo start testing
testing start/running, process 27794

Now if I kill the PID of the above process, then it get restarted automatically within few milliseconds or microseconds.

Is there any possibility of restarting my python script after 1 second if it has gone down or got killed? I don't want to restart immediately.

Answer 1

This is completely untested, but it's too complex to fit in a comment, so hopefully it works.

You might be able to add a post-stop stanza which will do what you want. Something such as:

post-stop script
  [ "$RESULT" == "failed" ] && sleep 1
end script

(there is also a $EXIT_STATUS you can use)

For documentation see the upstart cookbook sections post-stop and environment variables.

The cookbook even has a section on "delay respawn of a job", but I think the documentation may be incorrect. It says it'll only delay with a non-zero exit code, but there is nothing checking the exit code in their example, so it would run unconditionally, which isn't what you want.

Answer 2

You can use the next post-stop script for sleep only before respawn job (without sleep when just 'stop' or 'restart'):

post-stop script
  # Change value of STIME to your number of seconds:
  STIME=15
  initctl status $UPSTART_JOB |grep 'start/post-stop' >/dev/null && ( echo -n `date`; echo ": Going to sleep of $STIME seconds before respawn..."; sleep $STIME ) || true
end script

It is work correctly for me.

Answer 3

You are going to have to add a line:

post-stop exec sleep 1

This, unfortunately, causes the stop action to always take a second longer, but it is the only solution (the RESULT variable is only for the event, not the exec sections; it will not be available like Patrick shows).

Edit: ooh ooh I got something. You can create a new job, say testing-wait-respawn, with the following contents to get it only when the job has failed:

start on stopping testing RESULT=failed

task

exec sleep 1

Answer 4

A good upstart documentation on ubuntu.com: Delay respawn of a job

respawn
exec mydaemon
post-stop exec sleep 1