I have a Python / JS server combo that I need to keep running at all times. The Python server has a tendency to crash, as it's running on a server with 1GB of RAM.
I've written a script called "monitor.sh" which is set in crontab to run every minute, but it seems to fail whenever it tries to restart the python server.
I know that it is running, I can verify it is running by the timestamp it leaves every time it runs.
monitor.sh
#!/bin/bash
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/www/site.com
echo "CHECKING" >> /var/www/site.com/monitor.log
pidof python3 >/dev/null
if [ $? -ne 0 ] ; then
echo "Restarting Python App: $(date)" >> /var/www/site.com/monitor.log
screen -dmS python_server python3 /var/www/site.com/app.py
fi
pidof nodejs >/dev/null
if [ $? -ne 0 ] ; then
echo "Restarting Inferno Server: $(date)" >> /var/www/site.com/monitor.log
screen -dmS inferno_server nodejs /var/www/site.com/render_server.js
fi
crontab -- (crontab -e under the typical user)
* * * * * /var/www/site.com/monitor.sh
I will see "Restarting Python App" in the logs, and then nothing will happen. Attempting to log the output with > out.txt 2> errors.txt produces nothing.
Things that have NOT worked so far:
- Using the system level crontab
- Running the server with sudo
- Using
chmod +xon all files - Using
chmod -R 755on all files - Calling
/usr/bin/python3directly - Using
os.chdir()from within the python script
Things that DO work:
- Running the app manually
- Running the script manually
- Running the script with any other Python file
- Monitoring and re-invoking the other JS server as necessary
The server is a bottle.py instance using Gunicorn.
Is there any way I can find out what's actually happening here? A log file I can check, or a way to actually get errors out of this command?
screen? Do you have to interact with its console as it runs? Also, it might be better to check the results ofscreen -lsrather than looking for anypython3process owned by any user. – DopeGhoti Nov 28 '16 at 21:46