0

I wrote a python script that I want to keep running indefinitely. Most of the time it sleeps, but every once in a while it has to come to life to do some operations. I need to keep it running on an EC2 instance and I need to disconnect from the terminal when I do it.

I thought this would be the ideal solution:

#!/bin/bash

while true; do
  python monitor.py >>monitor.log 2>&1
  echo "monitor.py exited with code $?. Respawning.." >&2
  sleep 2
done

I run this file, saved as monitor.sh, via:

./monitor.sh >>monitor.log 2>&1 &

If I come back a day or two later and run:

ps aux | grep monitor

I usually see both monitor.sh and monitor.py in the output, but not always. Once in a while I will see neither. So something has happened to both of those processes.

The expected behaviour is that if monitor.py exits for some reason, either with exit code 0 or otherwise, then the echoed line should appear in a log file and the python script should restart.

Why doesn't it? What could be happening here that my script sometimes stalls.

Notice that I put a & at the end of the shell script when I run it in order to disconnect from the terminal which I then log out of. Could that be part of the problem?

I never see the results of the echo "monitor.py exited with code $?. Respawning.." >&2 in the log file unless I specifically kill it myself (with kill $pid_of_monitor.py). When left to run its own course, there is never a hint that the python script failed in the log files. I'm not sure what to conclude from that.

  • 2
    Are you using GNU screen or tmux to run the shell session? If not, the shell may be sending a HUP signal to the jobs when it's terminating. Did you try running the script with nohup or use disown on the backgrounded job? – Kusalananda Feb 5 at 21:53
  • @Kusalananda, HUP may have something to do with it. I'm not 100% sure, but I seem to be able to reproduce the pattern if I manually kill the processes with SIGHUP. I will try using nohup for a few days and report back what I find. – Octopus Feb 5 at 22:04
  • I remain unconvinced at this time though, why would a HUP signal take so long to affect those procs? They usually run for 20+ hours after I logout before bailing. – Octopus Feb 5 at 22:14
  • Is there something on this system that auto-kills long running jobs, possibly when they reach a certain amount of CPU time? Why do you need to respawn your Python program? – Kusalananda Feb 5 at 23:04

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.