How can I scp files during a systemd shutdown?

Question

My system has certain files which I need to transfer using scp as a node is shutting down.

I have a systemd service file which looks like this.

[Unit]
Description=My Transfer Service
After=multi-user.target sockets.target network.target auditd.service \
    timers.target sshd.socket [email protected] systemd-user-sessions.service \
    [email protected] system-sysctl.service paths.target swap.target \
    network-online.target

[Service]
Type=oneshot
ExecStart=/bin/echo "My Transfer Service"
ExecStop=/custom/dir/transfer.bash
RemainAfterExit=yes
TimeoutStopSec=0

[Install]
WantedBy=default.target timers.target

The transfer script looks like this

#!/bin/bash

scp -r /important/dir otheruser@target_node:~/
if [ "$?" != "0" ]
then
    echo "ERROR `date`: Failure to transfer" >> transfer_log
    exit 1
fi

echo "SUCCESS `date`" >> transfer_log

The ssh keys are set up correctly between *this* system's root user and the other system's "other_user". This scp does not require a password.

The transfer succeeds if I run the script standalone as root user.

The transfer also succeeds if I use the command systemctl stop my-transfer

The transfer will fail when I call reboot. When I log back in I see error messages in my log.

How can I use systemd to scp files during a shutdown? (This is a RedHat distro)

Edit:

The error that scp gives me is

ssh_exchange_identification: Connection closed by remote host
lost connection

My serial output up until the transfer service stops

         Stopping Load/Save Random Seed...
         Stopping Load/Save Screen Backlight...ness of backlight:acpi_video0...
[  OK  ] Stopped target System Time Synchr[  OK  ] Stopped Session c5 of user r.
         Stopping My-Transfer Service...
[  OK  ] Stopped Network Time Synchroni[  OK  ] Stopped Serial Getty on ttyS0.
[  OK  ] Stopped Load/Save Random Seed.
[  OK  ] Stopped Load/Save Screen Backlight ...htness of backlight:acpi_video0.
[  OK  ] Stopped My-Transfer Service.

Update

The transfer succeeds on shutdown if a regular cp command is used. That doesn't fix my problem, but it does prove the error is definitely related to scp somehow.

The command systemctl status my-transfer-service shows my service is active (exited). The script transfer.bash is also usually run by crond at various intervals during system operation. Could it be that cron is attempting to run its jobs at shutdown as well?

Answer 1

Missing from your [Unit] is the magic line:

Conflicts=reboot.target

This should cause your stop script to be called before the reboot service can be started and the network dies.

Answer 2

I'll post this as an answer since it's what I did, and I found it to be much easier than trying to shove down every systemd process I could find.

I created a special shutdown script to perform the actions before executing either reboot or shutdown. I aliased shutdown to sudo /my/special/script.sh, and followed a similar pattern for reboot , as well as aliasing alias sudo = 'sudo '. I also had to add this script to the sudoers files for the appropriate users.