I wrote a python script that I want to keep running indefinitely. Most of the time it sleeps, but every once in a while it has to come to life to do some operations. I need to keep it running on an EC2 instance and I need to disconnect from the terminal when I do it.
I thought this would be the ideal solution:
#!/bin/bash while true; do python monitor.py >>monitor.log 2>&1 echo "monitor.py exited with code $?. Respawning.." >&2 sleep 2 done
I run this file, saved as
./monitor.sh >>monitor.log 2>&1 &
If I come back a day or two later and run:
ps aux | grep monitor
I usually see both monitor.sh and monitor.py in the output, but not always. Once in a while I will see neither. So something has happened to both of those processes.
The expected behaviour is that if monitor.py exits for some reason, either with exit code 0 or otherwise, then the echoed line should appear in a log file and the python script should restart.
Why doesn't it? What could be happening here that my script sometimes stalls.
Notice that I put a
& at the end of the shell script when I run it in order to disconnect from the terminal which I then log out of. Could that be part of the problem?
I never see the results of the
echo "monitor.py exited with code $?. Respawning.." >&2 in the log file unless I specifically kill it myself (with
kill $pid_of_monitor.py). When left to run its own course, there is never a hint that the python script failed in the log files. I'm not sure what to conclude from that.