1

I have a shell script which runs three programs in the background, a few in the foreground and then trap and wait, and I have set up a unit file so systemd can start it and restart it if it fails.

I've found, however, that if one process dies, it doesn't kill everything in that script and restart it. For this application they must all be restarted if any one of them dies.

I see two reasonable paths:

  1. Configure the unit file and possibly change the script so it detects that abnormality and kills them all, rerunning the script. I'm not sure how to do this.
  2. Configure each of the three background processes as its own unit with individual .service files. But I don't know how to write the .service file to kill and restart each of them if any one of them fails. I know I can arrange their dependency so they start in order, but I don't know how to have it kill #1 when #2 dies, or vice versa.

I don't want to write a manager or have the programs figure it out and die themselves - that's what systemd is for - I expect I'm just missing the right incantation.

.service file:

[Unit]
Description=Foobar Interface
After=network.target

[Service]
Type=simple
WorkingDirectory=/home/user/scripts
ExecStart=/home/user/scripts/myscript.sh
Restart=always

[Install]
WantedBy=multi-user.target

Bash script:

#!/usr/bin/env bash

tty_port=/dev/ttyUSB0

#Clean up any old running processes
pkill -f "cat ${tty_port}"
pkill transport
pkill backgroundprogram

#Configure the target
source /home/user/somescript.sh
foregroundprogram

#Set up the serial port
stty -F $tty_port 115200 

#Read from the port in the background
cat $tty_port &
tty_pid=$!

#Wait for tty device to waken
sleep 15

#Send commands to tty device
echo "command1" > $tty_port
sleep 1
echo "command2" > $tty_port
sleep 1

#Start up the transport
/home/user/transport &>> /dev/null &
transport_pid=$!

#Wait a bit for the transport to start
sleep 1

#Start up the main process
/home/user/backgroundprogram &
background_pid=$!

#Wait a bit for it to start
sleep 1

#Finally, start the tty device
echo "command3" > $tty_port

trap "kill ${background_pid} ${tty_pid} ${transport_pid}; exit 1" INT
wait

It all works, writes to the journal, etc, but when any one of the three processes fails, it keeps chugging along and doesn't kill and restart everything.

2

I've found, however, that if one process dies, it doesn't kill everything in that script and restart it. For this application they must all be restarted if any one of them dies.

systemd is monitoring your shell script, not it's children. You wouldn't want systemd to respond to children exiting, because that would result in a restart every time you ran a command. Consider, if I have a shell script that runs...

date

I just spawned a child process, it ran, and then exited. I don't want this to trigger any action from my process supervisor.

If you want systemd to monitor the child processes, then create a separate unit file for each process:

  1. One unit for configuring and reading from the serial port
  2. One for /home/user/transport
  3. One for /home/user/backgroundprogram

You can use systemd dependencies to ensure the correct start order of the services (and to ensure that if you stop one they all stop), and you could use EnvironmentFile directives to load configuration (like $tty_port) from a file.

You would probably put some of your setup commands ("Send commands to tty device...") into an ExecStartPre line, or maybe they would get their own Type=oneshot service.

  • 1
    How would I configure the transport unit .service file to stop and restart transport if backgroundprogram fails, and vice versa? – Adam Davis Feb 8 at 21:06
2

If you can split your main script into separate services, you can easily solve it like this:

In the following example I have three respawning services, s1, s2 and s3, and control them all as a group via the target s.target.

Note:
If you configure the three services as Requires in your s.target, then if one of them crash and respawn, all participating processes in this group are restarted.
Or if you configure them as Wants in your s.target, then if one of them crash and respawn, only this individual processes is restarted.




For each service, create a service file s1, s2, s3:

/etc/systemd/system/s1.service :

[Unit]
Description=my worker s1
After=network.target
Before=foobar.service
PartOf=s.target

[Service]
Type=simple
ExecStart=/usr/local/bin/s1.sh
Restart=always

(note: if your services are identical you can create one s1@.service file instead of multiple files. Look in the manual for service instances using @ and %i.)


Now create the main target (group) file which requires the s1, s2 and s3 services:

/etc/systemd/system/s.target :

[Unit]
Description=main s service
Requires=s1.service s2.service s3.service
# or
# Wants=s1.service s2.service s3.service

[Install]
WantedBy=multi-user.target

Done.
As usual you must now run systemctl daemon-reload.

Now you can start your services with systemctl start s.target
s1, s2 and s3 are started.

You can stop all three services with systemctl stop s.target
s1, s2 and s3 are stopped.

Naturally you can start/stop/restart/status the individual services as per usual:
systemctl status s1

If you kill the s1, s2 or s3 process, it will automatically respawn (Restart=always).
If you used Requires, then all processes in the group will be restarted.

PS: Run systemctl enable s.target if you want to start the services at boot.

PS: Unfortunetly, when using systemctl you can't use the shorthand word "s" for "s.target" as you can do with "s1" instead of typing the full "s1.service". You have to type "s.target" when you want to manage the group.

1
#!/usr/bin/env/python3
# POSIX shell and bash < 4.3 doesn't want to do this.
# https://unix.stackexchange.com/questions/285156/exiting-a-shell-script-if-certain-child-processes-exit
#
# If you haven't written python3 before, be aware the string type
# is Unicode (UTF-8).  Python 3.0 aborts on invalid UTF-8.
# Python 3.1 aims to round-trip invalid UTF-8 using "surrogateescape".
# Python 3.2 may accept non-UTF-8 encoding according to your locale.
# ...
#
# * Functions should be better tested.
#
# * Doesn't bother killing (and waiting for) child processes.
#   Assumes systemd does it for us.
#   Convenient, but I'm not 100% happy about it.
#
# * Otherwise direct translation of nasty script, e.g. use of "sleep".

import sys
import os
import time

tty_port = "/dev/ttyS0"  # or: tty_port = sys.environ["tty_port"]

def die(msg):
    sys.exit(msg)

# Run program in background
def bg(*argv):
    pid = os.fork()
    if pid == 0:
        # Child process: exec or die
        # Either way, we never return from this function.
        try:
            os.execvp(argv[0], argv)
        except Exception as e:
            # By convention, child always uses _exit()
            sys._exit(e)
        assert False
    return pid

def __fg(*argv):
    pid = bg(*argv)
    (_, status) = os.waitpid(pid, 0)
    return status

# Run program, wait for exit, die if the program fails
def fg(*argv):
    status = __fg(*argv)
    if os.WIFEXITED(status):
        code = os.WEXITSTATUS(status)
        if code != 0:
            die("exit status {} from running {}".format(code, argv))
    elif os.WIFSIGNALED(status):
        die("signal {} when running {}"
            .format(os.WTERMSIG(status), argv))
    else:
        assert False, "Unexpected result from waitpid()"

# Use with care.
# "Any  user input that is employed as part of command should be carefully sanitized, to ensure that unexpected shell commands or command options are  not executed."
#
def bg_shell(cmd):
    return bg("/bin/sh", "-c", cmd)

def fg_shell(cmd):
    return fg("/bin/sh", "-c", cmd)


fg("stty", "-F", tty_port, "115200")

tty_pid = bg("cat", tty_port)
print("\"cat {}\" started as pid {}".format(tty_port, tty_pid))

time.sleep(15)

tty_out = open(tty_port, "w")
def tty_print(msg):
    tty_out.write(msg)
    tty_out.flush()

tty_print("command1")
time.sleep(1)

tty_print("command2")
time.sleep(1)

transport_pid = bg_shell("exec /home/user/transport >/dev/null 2>&1")
print("transport started as pid {}".format(transport_pid))
time.sleep(1)

tty_print("command3")
time.sleep(1)

background_pid = bg("/home/user/backgroundprogram")
print("backgroundprogam started as pid {}".format(background_pid))

(pid, status) = os.wait()

# This could be modified to accept exit code 0 as a success,
# and/or accept exit due to SIGTERM as a success.

if os.WIFEXITED(status):
    die("exit status {} from pid {}".format(os.WEXITSTATUS(status)), pid)
elif os.WIFSIGNALED(status):
    die("signal {} when running {}".format(os.WTERMSIG(status), pid))
else:
    assert False, "Unexpected result from wait()"
1

In recent versions of bash the wait command has an option -n which waits until any background process ends, then exits.

Additionally, for an as yet unknown reason, cat was occasionally exiting between being started and wait, but wasn't announced as ending until wait. So I added a jobs command immediately before wait, whcih seems to check to see if cat has exited or not. If it has, wait only pays attention to the two remaining processes. If it hasn't exited, then wait ends when any one of the three processes ends.

So the last wait line in my script is replaced with

jobs
wait -n

If any of the jobs ends after wait is called, then wait exits, systemd kills any remaining child processes, and restarts the script.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.