Monitor process and restart when not running using crontab

Question

I have a Python / JS server combo that I need to keep running at all times. The Python server has a tendency to crash, as it's running on a server with 1GB of RAM.

I've written a script called "monitor.sh" which is set in crontab to run every minute, but it seems to fail whenever it tries to restart the python server.

I know that it is running, I can verify it is running by the timestamp it leaves every time it runs.

monitor.sh

#!/bin/bash
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/www/site.com

echo "CHECKING" >> /var/www/site.com/monitor.log

pidof python3 >/dev/null
if [ $? -ne 0 ] ; then
  echo "Restarting Python App:  $(date)" >> /var/www/site.com/monitor.log
  screen -dmS python_server python3 /var/www/site.com/app.py
fi

pidof nodejs >/dev/null
if [ $? -ne 0 ] ; then
  echo "Restarting Inferno Server:  $(date)" >> /var/www/site.com/monitor.log
  screen -dmS inferno_server nodejs /var/www/site.com/render_server.js
fi

crontab -- (crontab -e under the typical user)

* * * * * /var/www/site.com/monitor.sh

I will see "Restarting Python App" in the logs, and then nothing will happen. Attempting to log the output with > out.txt 2> errors.txt produces nothing.

Things that have NOT worked so far:

• Using the system level crontab
• Running the server with sudo
• Using chmod +x on all files
• Using chmod -R 755 on all files
• Calling /usr/bin/python3 directly
• Using os.chdir() from within the python script

Things that DO work:

• Running the app manually
• Running the script manually
• Running the script with any other Python file
• Monitoring and re-invoking the other JS server as necessary

The server is a bottle.py instance using Gunicorn.

Is there any way I can find out what's actually happening here? A log file I can check, or a way to actually get errors out of this command?

Answer 1

I know that's not what you're asking, but your approach seems too much tricky. I'm not even sure screen is supposed to work in a script and I don't have my pc with me right now to test it

What I strongly suggest you is to use supervisord to monitor and restart the services.

Answer 2

I would suggest using Upstart or Systemd. Depending on the age of your distribution Ubuntu/Debian/CentOS would have them pre-installed. Much older systems would have System V.