I have a Python / JS server combo that I need to keep running at all times. The Python server has a tendency to crash, as it's running on a server with 1GB of RAM.
I've written a script called "monitor.sh" which is set in crontab to run every minute, but it seems to fail whenever it tries to restart the python server.
I know that it is running, I can verify it is running by the timestamp it leaves every time it runs.
monitor.sh
#!/bin/bash
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/www/site.com
echo "CHECKING" >> /var/www/site.com/monitor.log
pidof python3 >/dev/null
if [ $? -ne 0 ] ; then
echo "Restarting Python App: $(date)" >> /var/www/site.com/monitor.log
screen -dmS python_server python3 /var/www/site.com/app.py
fi
pidof nodejs >/dev/null
if [ $? -ne 0 ] ; then
echo "Restarting Inferno Server: $(date)" >> /var/www/site.com/monitor.log
screen -dmS inferno_server nodejs /var/www/site.com/render_server.js
fi
crontab -- (crontab -e
under the typical user)
* * * * * /var/www/site.com/monitor.sh
I will see "Restarting Python App" in the logs, and then nothing will happen. Attempting to log the output with > out.txt 2> errors.txt
produces nothing.
Things that have NOT worked so far:
- Using the system level crontab
- Running the server with sudo
- Using
chmod +x
on all files - Using
chmod -R 755
on all files - Calling
/usr/bin/python3
directly - Using
os.chdir()
from within the python script
Things that DO work:
- Running the app manually
- Running the script manually
- Running the script with any other Python file
- Monitoring and re-invoking the other JS server as necessary
The server is a bottle.py instance using Gunicorn.
Is there any way I can find out what's actually happening here? A log file I can check, or a way to actually get errors out of this command?
screen
? Do you have to interact with its console as it runs? Also, it might be better to check the results ofscreen -ls
rather than looking for anypython3
process owned by any user.