I have an Upstart script that uses the following code:
start on runlevel [2345]
stop on runlevel [!2345]
respawn
env PYTHONPATH=$PYTHONPATH:/opt/lrms/lib/python2.6/site-packages
exec python /opt/foo_service/src/foo/foo.py >> /var/log/foo.log 2>&1
pre-start script
echo "[`date`] - Starting Foo Service..." >> /var/log/foo.log
end script
pre-stop script
echo "[`date`] - Stopping Foo Service..." >> /var/log/foo.log
end script
post-stop script
echo "[`date`] - Post-Stop Stanza..." >> /var/log/foo.log
end script
I can easily start and stop this service using start foo or stop foo. However, when I reboot the server and check the status of my service, it is in stop/waiting.
I then check the logs, and see this (these are the events that took place with this upstart script during startup):
[Mon Aug 22 12:20:34 EDT 2016] - Starting Foo Service...
[Mon Aug 22 12:20:35 EDT 2016] - Post-Stop Stanza...
[Mon Aug 22 12:20:35 EDT 2016] - Starting Foo Service...
[Mon Aug 22 12:20:35 EDT 2016] - Post-Stop Stanza...
[Mon Aug 22 12:20:35 EDT 2016] - Starting Foo Service...
[Mon Aug 22 12:20:35 EDT 2016] - Post-Stop Stanza...
[Mon Aug 22 12:20:35 EDT 2016] - Starting Foo Service...
[Mon Aug 22 12:20:36 EDT 2016] - Post-Stop Stanza...
[Mon Aug 22 12:20:36 EDT 2016] - Starting Foo Service...
[Mon Aug 22 12:20:36 EDT 2016] - Post-Stop Stanza...
[Mon Aug 22 12:20:36 EDT 2016] - Starting Foo Service...
[Mon Aug 22 12:20:36 EDT 2016] - Post-Stop Stanza...
[Mon Aug 22 12:20:36 EDT 2016] - Starting Foo Service...
[Mon Aug 22 12:20:36 EDT 2016] - Post-Stop Stanza...
[Mon Aug 22 12:20:36 EDT 2016] - Starting Foo Service...
[Mon Aug 22 12:20:36 EDT 2016] - Post-Stop Stanza...
[Mon Aug 22 12:20:36 EDT 2016] - Starting Foo Service...
[Mon Aug 22 12:20:37 EDT 2016] - Post-Stop Stanza...
[Mon Aug 22 12:20:37 EDT 2016] - Starting Foo Service...
[Mon Aug 22 12:20:37 EDT 2016] - Post-Stop Stanza...
[Mon Aug 22 12:20:37 EDT 2016] - Starting Foo Service...
[Mon Aug 22 12:20:37 EDT 2016] - Post-Stop Stanza...
This implies to me that the upstart script is crashing, considering the pre-stop stanza is not being executed, and it respawned multiple times.
My next guess is that the Python script could depend on something that isn't currently running when the upstart script is starting. The Python service captures events from ActiveMQ and stores them into MySQL. I've tried editing the Upstart script so it runs when MySQL is ready, but this didn't output anything to the log file at all.
Am I correct in assuming that this is probably an issue with the Python service depending on some other service running? How should I go about debugging this issue?
