Preventing multiple python scripts from crashing with single shell script?

Question

I would like to create a shell script to prevent a python script from crashing in a Raspberry Pi. This python script is auto started on reboot with cron, however, sometimes it can crash while running.

Currently my shell script reads:

#!/bin/sh

COMMAND='python home/pi/projects/mypythonscript.py'
LOGFILE=restart.txt

writelog() {
  now=`date`
  echo "$now $*" >> $LOGFILE
}

writelog "Starting"
while true ; do
  $COMMAND
  writelog "Exited with status $?"
  writelog "Restarting"
done

I wanted to ask if I can create two command lines, in order to make sure two independent scripts can be prevented from crashing with this shell script? Or do I have to make a new shell script file for the second python script?

I am thinking of modifying it this way:

#!/bin/sh

COMMAND1='python home/pi/projects/mypythonscript1.py'
COMMAND2='python home/pi/projects/mypythonscript2.py'
LOGFILE=restart.txt

writelog() {
  now=`date`
  echo "$now $*" >> $LOGFILE
}

writelog "Starting"
while true ; do
  $COMMAND1
  $COMMAND2
  writelog "Exited with status $?"
  writelog "Restarting"
  done

Would this modification work? I appreciate any advice as I am still quite new to the linux (debian) platform.

Answer 1

If you are using systemd you could create a service for each command, then get systemd to restart it if it crashes.

Something like:

[Unit]
Description='description of script'

[Service]
ExecStart=/path/too/script
Restart=always

[Install]
WantedBy=multi-user.target

This can then be put into /etc/systemd/system, after running systemctl daemon-reload you will be able to start the service. It will also start after a reboot.

Answer 2

If you want to handle an arbitrary number of asynchronous processes from one control script, you can use this approach:

#!/bin/sh
COMMAND1=(python home/pi/projects/mypythonscript1.py)
COMMAND2=(python home/pi/projects/mypythonscript2.py)
    ︙
rm -f COMMAND1_failed; ("${COMMAND1[@]}"; touch COMMAND1_failed)&
rm -f COMMAND2_failed; ("${COMMAND2[@]}"; touch COMMAND2_failed)&
       ︙
while true
do
        if [ -e COMMAND1_failed ]
        then
                # Restart Command1
                rm -f COMMAND1_failed; ("${COMMAND1[@]}"; touch COMMAND1_failed)&
        fi
        if [ -e COMMAND2_failed ]
        then
                # Restart Command2
                rm -f COMMAND2_failed; ("${COMMAND2[@]}"; touch COMMAND2_failed)&
        fi
         ︙
        sleep 60
done

• COMMAND1=(python home/pi/projects/mypythonscript1.py) defines COMMAND1 to be an array.  This allows the command to include words that contain spaces; for example, something like COMMAND1=(python "home/pi/projects/my python script1.py").
• "${COMMAND1[@]}" is analogous to your use of $COMMAND1; it runs your first command, quoting its constituent words.
• (commandA; commandB) essentially creates an ad-hoc two-line script:

  commandA
  commandB

  so ("${COMMAND1[@]}"; touch COMMAND1_failed) says “run command #1, wait for it to finish, and then touch (create) a file called COMMAND1_failed.  This entire mini-script runs in the background, so the main script can go on and run command #2, etc.
• Realistically, you should probably use full pathnames for the failed files, and maybe put their names into variables.
• After starting all the background jobs, the main script enters an infinite (forever) loop.  If the COMMAND1_failed file exists, that means that command #1 and needs to be restarted.  Etc.
• Adjust the sleep as desired.  Too low a value, and the control script might run so often that it affects the performance of the system.  Too high a value and the asynchronous tasks will not be restarted promptly, so the responsiveness of the system would suffer in that way.

The above starts all the processes, and then enters an infinite loop.  Here is a trivial optimization:

#!/bin/sh
    ︙
touch COMMAND1_failed
       ︙
while true
do
        if [ -e COMMAND1_failed ]
        then
                # (Re)start Command1
                rm -f COMMAND1_failed; ("${COMMAND1[@]}"; touch COMMAND1_failed)&
        fi
         ︙
        sleep 60
done

which starts the background processes on the first iteration of the loop, after creating all the necessary COMMANDn_failed files to make it look like the processes need to be (re)started.  The only difference is that it lists the commands to start the processes once instead of twice.

See also:

• Group a set of processes such that when one dies, so do all the other processes in that group.  In particular, Kamil Maciorowski’s answer partially inspired my answer here.
• How should I check whether a given PID is running?
• Running a script when background process crashes
• Given two background commands, terminate the remaining one when either exits