0

I have a Python / JS server combo that I need to keep running at all times. The Python server has a tendency to crash, as it's running on a server with 1GB of RAM.

I've written a script called "monitor.sh" which is set in crontab to run every minute, but it seems to fail whenever it tries to restart the python server.

I know that it is running, I can verify it is running by the timestamp it leaves every time it runs.

monitor.sh

#!/bin/bash
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/www/site.com

echo "CHECKING" >> /var/www/site.com/monitor.log

pidof python3 >/dev/null
if [ $? -ne 0 ] ; then
  echo "Restarting Python App:  $(date)" >> /var/www/site.com/monitor.log
  screen -dmS python_server python3 /var/www/site.com/app.py
fi

pidof nodejs >/dev/null
if [ $? -ne 0 ] ; then
  echo "Restarting Inferno Server:  $(date)" >> /var/www/site.com/monitor.log
  screen -dmS inferno_server nodejs /var/www/site.com/render_server.js
fi

crontab -- (crontab -e under the typical user)

* * * * * /var/www/site.com/monitor.sh

I will see "Restarting Python App" in the logs, and then nothing will happen. Attempting to log the output with > out.txt 2> errors.txt produces nothing.

Things that have NOT worked so far:

  • Using the system level crontab
  • Running the server with sudo
  • Using chmod +x on all files
  • Using chmod -R 755 on all files
  • Calling /usr/bin/python3 directly
  • Using os.chdir() from within the python script

Things that DO work:

  • Running the app manually
  • Running the script manually
  • Running the script with any other Python file
  • Monitoring and re-invoking the other JS server as necessary

The server is a bottle.py instance using Gunicorn.

Is there any way I can find out what's actually happening here? A log file I can check, or a way to actually get errors out of this command?

  • Have you tried setting it up the in crontab of the actual user which typically runs the process? – DopeGhoti Nov 28 '16 at 21:25
  • That's how it's currently setup, as a line in 'crontab -e' under the typical user. No dice, unfortunately. – Noi Sek Nov 28 '16 at 21:26
  • Is there a reason you need to run it in screen? Do you have to interact with its console as it runs? Also, it might be better to check the results of screen -ls rather than looking for any python3 process owned by any user. – DopeGhoti Nov 28 '16 at 21:46
0

I know that's not what you're asking, but your approach seems too much tricky. I'm not even sure screen is supposed to work in a script and I don't have my pc with me right now to test it

What I strongly suggest you is to use supervisord to monitor and restart the services.

  • I'm not sure that it actually works, but this seems like a better (or at least functional) solution. – Noi Sek Nov 29 '16 at 1:09
0

I would suggest using Upstart or Systemd. Depending on the age of your distribution Ubuntu/Debian/CentOS would have them pre-installed. Much older systems would have System V.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.