Stop rsync scheduled task race condition (large directory, small time interval)

Question

Platform information:

OpenBSD 6.2 amd64

$ rsync --version rsync version 3.1.2 protocol version 31

I'm trying to sync a large directory (4TB) using the following daily.local file (for Linux admins, this is essentially a cron daily task):

#!/bin/sh
# Sync the primary storage device to the backup disk
/usr/local/bin/rsync -avz /mnt/media_primary/ /mnt/media_backup/

The initial rsync copy takes more than a day. After a day or two, I end up with multiple running copies of rsync in my processes list: new ones are started as scheduled and these new processes seem to be competing with each other and not finishing the task (quickly at least)!

Is there a way to make a new rsync process aware of other rsync processes (or another way to avoid rsync race conditions)?

I know I could just run rsync manually to copy over the directory the first time and/or increase the scheduled time interval. This question is more for my interest as I was unable to find information on the net about this topic.

Answer 1

A solution that would be somewhat similar to Mark's solution, but without requiring changes to the /etc/fstab or mounting and unmounting filesystems:

#!/bin/sh

lockdir=/tmp/file-copy.lock

if ! mkdir "$lockdir"; then
   echo 'File copy already in progress' >&2
   exit 1
fi

trap 'rmdir "$lockdir"' EXIT

PATH=$PATH:/usr/local/bin

rsync -ai ...

A few notes about this:

1. The mkdir is an atomic operation, much like mount, that will fail if the directory already exists, but create the directory if it doesn't. This is safer than first checking for a lock file and then creating it (two steps with a possibility of a race condition in-between).

2. The EXIT trap ensures that the lock directory is deleted when the script terminates. The lock directory would also be deleted on reboots (by the system) since it's in /tmp.

3. I set PATH to the appropriate value rather than calling rsync with its full path. This is purely a cosmetic thing, but could be useful if the script later is expanded to use other commands from the OpenBSD ports collection (such as restic or borgbackup).

4. The -z option to rsync is really only needed over very slow network connections (when compressing/decompressing data is done quicker than the network bandwidth), and never for local copying. I also tend to favour the -i (--itemize-changes) over -v (--verbose), as this will tell me exactly why a file was transferred.

For backing up large amounts of data safely, I would generally recommend using a purpose written backup software over rsync, such as restic or borgbackup. These two additionally do data deduplication and encryption, and borgbackup may optionally do compression. restic is good in the sense that it allows one to save backups to an external server (over e.g. sftp), even if that server does not have restic installed, whereas borgbackup requires that the software is installed on the target system. Both restic and borgbackup handles backup repository locking.

Answer 2

One way round this problem (if the backup directory is on it's own partition) is to leave the volume unmounted, mounting just before starting the rsync command. This negates the need to use flock and may have the benefit of prolonging drive longevity/reducing power consumption.

/etc/fstab: add the noauto option to the partition so it doesn't get automatically mounted on boot

In the daily.local or cron.daily scheduled task:

#!/bin/sh
mount /mnt/media_backup && \
/usr/local/bin/rsync -avz /mnt/media_primary/ /mnt/media_backup/ && \
umount /mnt/media_backup

The double ampersand operator (&&) will only start the next command if the previous command is successful. Hence, if the backup disk can't be mounted (because it is already mounted and rsync is already running on the partition), the rest of the command will not proceed.