1

I'd like to run a little backup script (btrfs snapshot + transfer to usb) on system shutdown.

The following service executes perfectly fine on shutdown, but not on reboot as intended:

[Unit]
Description=Backup on poweroff to external encrypted USB disk.
DefaultDependencies=no
Before=shutdown.target

[Service]
Type=oneshot
ExecStart=/usr/bin/sleep 30
TimeoutSec=infinity
RemainAfterExit=yes

[Install]
WantedBy=poweroff.target

Where /usr/bin/sleep 30 will be replaced by the actual backup-script once this service works as intended.


But for the backup-script to work properly I need to decrypt an external USB disk with cryptsetup before it is started. This is all handled in the background by a combination of /etc/crypttab (noauto) and systemd (via systemd-cryptsetup-generator), which auto-generates a systemd unit-file systemd-cryptsetup@<name>.service that I can reference in the above unit.

But as soon as I add the following requirement to the [unit] section:

Requires=systemd-cryptsetup@ext.service
After=systemd-cryptsetup@ext.service

The service is not started anymore during shutdown/poweroff of the system.

Starting the service manually via systemctl start poweroff-backup.service works without a problem, so there is no issue with the service-file itself... :-/


I think the issue is that the poweroff.target via systemd-poweroff.service requires the umount.target which in turn unmounts all devices (including cryptsetup 'mounts') via a combination of ConflictedBy= and After= statements for the systemd mount und cryptsetup units. But adding another requirement towards the umount.target does not change anything, eg:

Before=umount.target

Without any of the additional requirements I can clearly see that the service is always started right after the umount.target has been reached.

1

Okay I found the issue and from that I was able to build a correctly working solution that stills follows the systemd workflow (for the most part).

Issue

The following service, while on first sight looking correct will never be executed:

# poweroff-backup.service
[Unit]
Description=Backup on poweroff to external encrypted USB disk.
DefaultDependencies=no
Before=shutdown.target

# This causes the service to be discarded/skipped due to
# a dependency cycle conflict. See below... 
Requires=systemd-cryptsetup@ext.service
After=systemd-cryptsetup@ext.service
Before=umount.target

[Service]
Type=oneshot
# Note:  The script executed here needs access to '/', '/tmp' 
# (or Systemd's PrivateTemp) and '/dev/mapper/ext' which is the 
# decrypted USB partition managed by /etc/crypttab and 
# systemd-cryptsetup-generator (as systemd-cryptsetup@ext.service)
ExecStart=/usr/bin/sleep 30
TimeoutSec=infinity
# Note: This does not make any difference
RemainAfterExit=yes

[Install]
WantedBy=poweroff.target

The reason why this service will never be started is because there is a dependency cycle issue/conflict between poweroff.target -[require]-> shutdown.target -[conflicts]-> cryptsetup.target <-[require]- systemd-cryptsetup@ext.service <-[require]- poweroff-backup.service <-[wants]- poweroff.target, where shutdown.target conflicts cryptsetup.target to ensure all encrypted disks are detached before shutdown (poweroff/reboot) and poweroff-backup.service which needs it to be activated by poweroff.target.

Solution

Knowing this, the obvious problem lies in my wrong assumption that systemd could somehow avoid this dependency cycle conflict by knowing that poweroff-backup.service is a "one-shot" service that can be resolved first by systemd while continuing with the normal shutdown transition once poweroff-backup.service has been activated and transitioned into inactive/done.

So here is a service that uses ExecStop= instead of ExecStart= which allows us to avoid any dependency-cycles, but has some minor caveats:

# poweroff-backup.service
[Unit]
Description=Backup external encrypted USB disk on poweroff.
Requires=systemd-cryptsetup@ext.service
After=multi-user.target
After=systemd-cryptsetup@ext.service

[Service]
Type=oneshot
ExecStart=/usr/bin/echo "Waiting for poweroff..."
# Note: We need this, since there is no other way to detect poweroff vs reboot now.
ExecStop=/usr/bin/systemctl list-jobs | /usr/bin/egrep -q 'poweroff.target.*start'
# Note: This should be replaced with your script
ExecStart=/usr/bin/sleep 30
TimeoutSec=infinity
RemainAfterExit=true

[Install]
WantedBy=multi-user.target

This works because:

  • It prevents the dependency cycle issue caused by shutdown.target, cryptsetup.target and poweroff-backup.service
  • The correct working order is still preserved, since systemd ensures that shutdown order is the reverse of the startup order (as given by After= and Before=)

There are only two minor gripes I have with this solution:

  • Since the service is started on login, it will also decrypt the external USB disk on login. I would prefer if this would only happen once the actual backup-script is executed while still using the normal systemd After= and Require= order and dependency mechanics. This could be handled by adding the following, but this would mean I'm not using the cryptsetup service for decrypting/detaching the USB disk:

    ExecStop=/usr/lib/systemd/systemd-cryptsetup attach ext
    [...]
    ExecStop=/usr/lib/systemd/systemd-cryptsetup detach ext
    
  • I don't have any way to conditionally execute ExecStop= only on shutdown, by referencing the shutdown.target and have to rely on a little helper command using systemctl list-jobs.

Question: Has somebody an idea how to improve on those points?

Full Service File

This is my current service file, which only starts the service once the USB disk is mounted, by attaching (WantedBy=) to the USB device (by its UUID) instead of the multi-user.target:

# poweroff-backup.service
[Unit]
Description=Backup external encrypted USB disk on poweroff.
Requires=systemd-cryptsetup@ext.service
Requires=-.mount
Requires=tmp.mount
After=dev-disk-by\<uuid here>.device
After=systemd-cryptsetup@ext.service
After=-.mount
After=tmp.mount

[Service]
Type=oneshot
ExecStart=/usr/bin/echo "Waiting for poweroff to trigger snapshot and archive transfer..."
ExecStop=/usr/bin/systemctl list-jobs | /usr/bin/egrep -q 'poweroff.target.*start'
ExecStop=-/usr/local/bin/btrfs-snapshots.sh --device='UUID=<uuid>' @ @boot @home
ExecStop=/usr/local/bin/btrfs-archive.sh --source='UUID=<uuid>' --target=/dev/mapper/ext
TimeoutSec=infinity
RemainAfterExit=true

[Install]
WantedBy=dev-disk-by\<uuid here>.device

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.