0

I wrote a python script that I want to keep running indefinitely. Most of the time it sleeps, but every once in a while it has to come to life to do some operations. I need to keep it running on an EC2 instance and I need to disconnect from the terminal when I do it.

I thought this would be the ideal solution:

#!/bin/bash

while true; do
  python monitor.py >>monitor.log 2>&1
  echo "monitor.py exited with code $?. Respawning.." >&2
  sleep 2
done

I run this file, saved as monitor.sh, via:

./monitor.sh >>monitor.log 2>&1 &

If I come back a day or two later and run:

ps aux | grep monitor

I usually see both monitor.sh and monitor.py in the output, but not always. Once in a while I will see neither. So something has happened to both of those processes.

The expected behaviour is that if monitor.py exits for some reason, either with exit code 0 or otherwise, then the echoed line should appear in a log file and the python script should restart.

Why doesn't it? What could be happening here that my script sometimes stalls.

Notice that I put a & at the end of the shell script when I run it in order to disconnect from the terminal which I then log out of. Could that be part of the problem?

I never see the results of the echo "monitor.py exited with code $?. Respawning.." >&2 in the log file unless I specifically kill it myself (with kill $pid_of_monitor.py). When left to run its own course, there is never a hint that the python script failed in the log files. I'm not sure what to conclude from that.

  • 2
    Are you using GNU screen or tmux to run the shell session? If not, the shell may be sending a HUP signal to the jobs when it's terminating. Did you try running the script with nohup or use disown on the backgrounded job? – Kusalananda Feb 5 at 21:53
  • @Kusalananda, HUP may have something to do with it. I'm not 100% sure, but I seem to be able to reproduce the pattern if I manually kill the processes with SIGHUP. I will try using nohup for a few days and report back what I find. – Octopus Feb 5 at 22:04
  • I remain unconvinced at this time though, why would a HUP signal take so long to affect those procs? They usually run for 20+ hours after I logout before bailing. – Octopus Feb 5 at 22:14
  • Is there something on this system that auto-kills long running jobs, possibly when they reach a certain amount of CPU time? Why do you need to respawn your Python program? – Kusalananda Feb 5 at 23:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.