0

Question

How do I manage a systemd-nspawn container as a systemd unit?

Setup

Install an image

# machinectl pull-raw --verify=no https://ftp.halifax.rwth-aachen.de/fedora/linux/releases/30/Cloud/x86_64/images/Fedora-Cloud-Base-30-1.2.x86_64.raw.xz

Discover the image name

# machinectl list-images
NAME                            TYPE RO  USAGE  CREATED                     MODIF
Fedora-Cloud-Base-30-1.2.x86_64 raw  no  891.6M Fri 2019-04-26 02:14:49 UTC Fri 2

1 images listed.

Start an interactive shell inside Fedora container

# systemd-nspawn -M Fedora-Cloud-Base-30-1.2.x86_64

Write /root/app.py python3 application that handles signals

# https://stackabuse.com/handling-unix-signals-in-python/
import signal
import os
import time
import sys

def terminateProcess(signalNumber, frame):
    print(f'received signal {signalNumber}')
    print ('exiting...')
    sys.exit()

def receiveSignal(signalNumber, frame):
    print(f'received signal {signalNumber}')
    return

if __name__ == '__main__':
    # register the signals to be caught
    signal.signal(signal.SIGHUP, receiveSignal)
    signal.signal(signal.SIGINT, terminateProcess)
    signal.signal(signal.SIGQUIT, receiveSignal)
    signal.signal(signal.SIGILL, receiveSignal)
    signal.signal(signal.SIGTRAP, receiveSignal)
    signal.signal(signal.SIGABRT, receiveSignal)
    signal.signal(signal.SIGBUS, receiveSignal)
    signal.signal(signal.SIGFPE, receiveSignal)
    #signal.signal(signal.SIGKILL, receiveSignal)
    signal.signal(signal.SIGUSR1, receiveSignal)
    signal.signal(signal.SIGSEGV, receiveSignal)
    signal.signal(signal.SIGUSR2, receiveSignal)
    signal.signal(signal.SIGPIPE, receiveSignal)
    signal.signal(signal.SIGALRM, receiveSignal)
    signal.signal(signal.SIGTERM, terminateProcess)

    # output current process id
    print(f'pid {os.getpid()}')

    # wait in an endless loop for signals 
    while True:
        time.sleep(1)

Exit container using key combination

Control + ]]]

app.service attempt 1

Write /etc/systemd/system/app.service unit file

[Service]
ExecStart=/usr/bin/systemd-nspawn --keep-unit -M Fedora-Cloud-Base-30-1.2.x86_64 python3 -u /root/app.py
SyslogIdentifier=%N
  • --keep-unit switch keeps systemd-nspawn and app.py in the system.slice/app.service cgroup
  • -u switch for unbuffered output
  • SyslogIdentifier using %N specifier for the string "app", that's the unit name without the suffix

Reload systemd daemon

# systemctl daemon-reload

In another terminal, continously follow log output with systemd-journald

# journalctl -f -u app.service

Start app.service unit

# systemctl start app.service

Stop app.service unit

# systemctl stop app.service

Observe logs

-- Logs begin at Fri 2019-08-23 16:58:11 UTC. --
Aug 23 17:26:42 srv0 systemd[1]: Started app.service.
Aug 23 17:26:42 srv0 app[12745]: Spawning container Fedora-Cloud-Base-30-1.2.x86_64 on /var/lib/machines/Fedora-Cloud-Base-30-1.2.x86_64.raw.
Aug 23 17:26:42 srv0 app[12745]: Press ^] three times within 1s to kill container.
Aug 23 17:26:42 srv0 app[12745]: Failed to create directory /tmp/nspawn-root-afZQoJ/sys/fs/selinux: Read-only file system
Aug 23 17:26:42 srv0 app[12745]: Failed to create directory /tmp/nspawn-root-afZQoJ/sys/fs/selinux: Read-only file system
Aug 23 17:26:42 srv0 app[12745]: pid 1
Aug 23 17:26:54 srv0 systemd[1]: Stopping app.service...
Aug 23 17:26:54 srv0 app[12745]: Container Fedora-Cloud-Base-30-1.2.x86_64 terminated by signal KILL.
Aug 23 17:26:54 srv0 systemd[1]: app.service: Main process exited, code=exited, status=1/FAILURE
Aug 23 17:26:54 srv0 systemd[1]: Stopped app.service.
Aug 23 17:26:54 srv0 systemd[1]: app.service: Unit entered failed state.
Aug 23 17:26:54 srv0 systemd[1]: app.service: Failed with result 'exit-code'.
Aug 23 17:26:54 srv0 systemd[1]: Stopped app.service.

systemd-nspawn is using SIGKILL instead of SIGTERM.

See the line Aug 23 17:26:54 srv0 app[12745]: Container Fedora-Cloud-Base-30-1.2.x86_64 terminated by signal KILL. I don't want to SIGKILL app.py, I want to SIGTERM it.

app.service attempt 2

Read a github issue

https://github.com/systemd/systemd/issues/7105#issuecomment-467491778

Use the -a/--as-pid2 switch

[Service]
ExecStart=/usr/bin/systemd-nspawn --keep-unit --as-pid2 -M Fedora-Cloud-Base-30-1.2.x86_64 python3 -u /root/app.py
SyslogIdentifier=%N

daemon-reload, start, stop

Observe logs

Aug 23 17:29:59 srv0 systemd[1]: Started app.service.
Aug 23 17:29:59 srv0 app[12841]: Spawning container Fedora-Cloud-Base-30-1.2.x86_64 on /var/lib/machines/Fedora-Cloud-Base-30-1.2.x86_64.raw.
Aug 23 17:29:59 srv0 app[12841]: Press ^] three times within 1s to kill container.
Aug 23 17:29:59 srv0 app[12841]: Failed to create directory /tmp/nspawn-root-jaGbcx/sys/fs/selinux: Read-only file system
Aug 23 17:29:59 srv0 app[12841]: Failed to create directory /tmp/nspawn-root-jaGbcx/sys/fs/selinux: Read-only file system
Aug 23 17:29:59 srv0 app[12841]: pid 2
Aug 23 17:30:06 srv0 systemd[1]: Stopping app.service...
Aug 23 17:30:06 srv0 app[12841]: Container Fedora-Cloud-Base-30-1.2.x86_64 terminated by signal KILL.
Aug 23 17:30:06 srv0 systemd[1]: app.service: Main process exited, code=exited, status=1/FAILURE
Aug 23 17:30:06 srv0 systemd[1]: Stopped app.service.
Aug 23 17:30:06 srv0 systemd[1]: app.service: Unit entered failed state.
Aug 23 17:30:06 srv0 systemd[1]: app.service: Failed with result 'exit-code'.

app.py is now running as pid 2! But still receiving a SIGKILL signal instead of SIGTERM.

1

Read github issue more

https://github.com/systemd/systemd/issues/7105#issuecomment-467491778

Use the --kill-signal flag

[Service]
ExecStart=/usr/bin/systemd-nspawn --keep-unit --kill-signal=SIGTERM -M Fedora-Cloud-Base-30-1.2.x86_64 python3 -u /root/app.py
SyslogIdentifier=%N

reload systemd, start app.service, and stop app.service

systemctl daemon-reload
systemctl start app.service
systemctl stop app.service

Observe logs

Aug 23 17:51:32 srv0 systemd[1]: Started app.service.
Aug 23 17:51:32 srv0 app[12994]: Spawning container Fedora-Cloud-Base-30-1.2.x86_64 on /var/lib/machines/Fedora-Cloud-Base-30-1.2.x86_64.raw.
Aug 23 17:51:32 srv0 app[12994]: Press ^] three times within 1s to kill container.
Aug 23 17:51:32 srv0 app[12994]: Failed to create directory /tmp/nspawn-root-71uVxm/sys/fs/selinux: Read-only file system
Aug 23 17:51:32 srv0 app[12994]: Failed to create directory /tmp/nspawn-root-71uVxm/sys/fs/selinux: Read-only file system
Aug 23 17:51:32 srv0 app[12994]: pid 1
Aug 23 17:51:35 srv0 app[12994]: Trying to halt container. Send SIGTERM again to trigger immediate termination.
Aug 23 17:51:35 srv0 app[12994]: received signal 15
Aug 23 17:51:35 srv0 app[12994]: exiting...
Aug 23 17:51:35 srv0 systemd[1]: Stopping app.service...
Aug 23 17:51:35 srv0 app[12994]: Container Fedora-Cloud-Base-30-1.2.x86_64 exited successfully.
Aug 23 17:51:35 srv0 systemd[1]: Stopped app.service.
Aug 23 17:51:35 srv0 systemd[1]: Stopped app.service.

See how the SIGTERM is being passed to app.py!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.