Rsync Daemon Max Connections Traffic Jam

Question

I prefer to use rsync daemon for all of my rync needs because it provides clean centralized management and saves on system resources. Accordingly, my /etc/rsyncd.conf contains multiple module entries.

My wrapper scripts for my actual rsync commands are all while loops that will instantly / repeatedly re-connect rsync in case of dropped connections.

Problem:
The max connections = 1 variable entry for each module is being read globally rather than individually per each module. Thereby causing @ERROR: max connections (1) reached -- try again later to take place (whichever rsync-daemon connects first gets the single available incorrectly global max connection = 1, causing all other connections to fail.. annoying ).

Without the max connections = 1, the while loop has the ability to spin up unlimited threads and consume unnecessary resources, hence a connections per module limit. Meanwhile the max connections = 1 has the accompanying per module file.lock per documentation.

This is my /etc/rsyncd.conf:

[home]
path = /home/username
list = yes
use chroot = false
strict modes = false
uid = root
gid = root
read only = yes
# Data source information
max connections = 1
lock file = /var/run/rsyncd-home.lock

[prod-bkup]
path = /media/username/external/Server-Backups/Prod/today
list = yes
use chroot = false
strict modes = false
uid = root
gid = root
# Don't allow to modify the source files
read only = yes
max connections = 1
lock file = /var/run/rsyncd-prod-bkup.lock

[test-bkup]
path = /media/username/external/Server-Backups/Test/today
list = yes
use chroot = false
strict modes = false
uid = root
gid = root
# Don't allow to modify the source files
read only = yes
max connections = 1
lock file = /var/run/rsyncd-test-bkup.lock

[VminRoot2]
path = /root/VDI-Files
list = yes
use chroot = false
strict modes = false
uid = root
gid = root
# Don't allow to modify the source files
read only = yes
max connections = 1
lock file = /var/run/rsyncd-VminRoot2.lock

This is an example of one of my rsync-daemon wrapper scripts:

#!/bin/sh
#
#
while [ 1 ]
do

   cputool --load-limit 7.5 -- nice -n -15 rsync -avxP --no-i-r --rsync-path="rsync" --log-file=/var/log/rsync-home.log --exclude 'snap'  --exclude 'lost+found' --exclude=".*" --exclude=".*/" 127.0.0.1::home /media/username/external/home-files-only && sync && echo 3 > /proc/sys/vm/drop_caches
   
    if [ "$?" = "0" ] ; then
        echo "rsync completed normally"
        exit
    else
        echo "Rsync failure. Backing off and retrying..."
        sleep 10
    fi
done

#end of shell script

Question
How can I get rid of the ERROR: max connections (1) reached -- try again later error?

Answer 1

The solution that I finally arrived at was to ensure that my while loop only executes once, if the connection has not been established yet and/or connection broken.

Note: My solutions, rather workarounds, should not be necessary given the correct according to documentation rsync-daemon configs, but it seems that despite having individually named .lock files for each modules, the first script with a while loop was somehow still somehow able to connect(rather block) the other .lock files from being used by the other respective scripts. (rsync bug?)

Solution 1 (quick & dirty):

flock -n <lock file> <script>

Or in my case, using this command to execute my cron job:

flock -n /var/run/rsyncd-home.lock /path/to/my_script.sh

caveat - this leaves your script vulnerable to stale lock files that may prevent execution on the next time interval.

Solution 2:
So, I used a bullet-proof method (so I think... I invite folks to correct my understanding, if need be)...

First, I did apt install procmail

From there I edited /usr/local/bin/backupscript.sh as follows:

#!/bin/bash
#
LOCK=/var/run/rsyncd-home.lock
remove_lock()
{
    rm -f "$LOCK"
}
another_instance()
{
    echo "There is another instance running, exiting"
    exit 1
}
lockfile -r 0 -l 3600 "$LOCK" || another_instance
trap remove_lock EXIT

#new using rsyncd & perpetual restart
while [ 1 ]
do
   cputool --load-limit 7.5 -- nice -n -15 rsync -avxP --no-i-r --rsync-path="rsync" --log-file=/var/log/rsync-home.log --exclude 'lost+found' --exclude=".*" --exclude=".*/" 127.0.0.1::home /media/username/external/home-files-only && sync && echo 3 > /proc/sys/vm/drop_caches
   
    if [ "$?" = "0" ] ; then
        echo "rsync completed normally"
        exit
    else
        echo "Rsync failure. Backing off and retrying..."
        sleep 10
    fi
done

#end of shell script

PRESTO:
The script will only connect to rsync daemon once, it will re-connect on dropped connections thanks to the while loop, and there is no danger of stale lock files interrupting my backup process at future intervals... (i.e. problem solved).