rsync while ignoring missing files. Problems with --exclude-from flag and invalid characters written to log

Question

I'd like to move new music from my seedbox to my local server on an hourly cron job. My local music is moved into new folders by a musicbrainz script as soon as it finishes syncing on my local machine, meaning the next sync job needs to only sync new files from the seedbox, ignoring removed files on the local server. Both machines are Linux.

My current solution is to use the rsync --exclude-from flag to read a sync log file, appended to by the previous rsync. It's mostly successful however it appears some characters are causing some directories & files to not match causing re-syncing. It appears as though it may be a UTF-8 issue, so I've tried adding: env LC_ALL=en_US.UTF-8 rsync... and various --iconv configurations however I still can't get a reliable sync, here's my script:

echo "Rsync My Artists"
#
rsync -vrz \
--exclude-from=/mnt/user/sync/music-myartists/__sync-log \
-e ssh \
xxxx@xxxx.xxx.xxx:/home37/xxxx/downloads/deluge-filters/my-artists/ /mnt/user/sync/music-myartists/ \
| grep -v ^sending \
| grep -v ^sent \
| grep -v ^total \
| grep -v 'receiving incremental file list' \
>> /mnt/user/sync/music-myartists/__sync-log
#
echo "Beets Moving My Artists"
#
docker exec beets beet -c /config/config-move-quiet.yaml import /sync/music-myartists
#
find /mnt/user/sync/music-myartists/ -not -name '__sync-log' -delete
################################

Here's an example of a __sync-log file getting rewritten with the same content: https://pastebin.com/FMSfwUU8

Any ideas?

Answer 1

Ok I worked it out, it had nothing to do with utf-8, rather filenames triggering one of rsyncs three wildcard characters '*', '?', and '[' . The '?' and '*' weren't causing issues, however the '[' introduces a character class, such as [a-z] or [[:alpha:]], which caused mismatches.

After writing rsyncs progress to the log i added:

sed -i 's/\[/\*/g' /mnt/user/sync/music-manual/__sync-log

To replace '[' with '*'

is there a neat way of including this in the rsync command?

See INCLUDE/EXCLUDE PATTERN RULES