0

I'd like to move new music from my seedbox to my local server on an hourly cron job. My local music is moved into new folders by a musicbrainz script as soon as it finishes syncing on my local machine, meaning the next sync job needs to only sync new files from the seedbox, ignoring removed files on the local server. Both machines are Linux.

My current solution is to use the rsync --exclude-from flag to read a sync log file, appended to by the previous rsync. It's mostly successful however it appears some characters are causing some directories & files to not match causing re-syncing. It appears as though it may be a UTF-8 issue, so I've tried adding: env LC_ALL=en_US.UTF-8 rsync... and various --iconv configurations however I still can't get a reliable sync, here's my script:

echo "Rsync My Artists"
#
rsync -vrz \
--exclude-from=/mnt/user/sync/music-myartists/__sync-log \
-e ssh \
xxxx@xxxx.xxx.xxx:/home37/xxxx/downloads/deluge-filters/my-artists/ /mnt/user/sync/music-myartists/ \
| grep -v ^sending \
| grep -v ^sent \
| grep -v ^total \
| grep -v 'receiving incremental file list' \
>> /mnt/user/sync/music-myartists/__sync-log
#
echo "Beets Moving My Artists"
#
docker exec beets beet -c /config/config-move-quiet.yaml import /sync/music-myartists
#
find /mnt/user/sync/music-myartists/ -not -name '__sync-log' -delete
################################

Here's an example of a __sync-log file getting rewritten with the same content: https://pastebin.com/FMSfwUU8

Any ideas?

  • I can't see the problem you're describing in that large log file on your pastebin. Perhaps you could cut out a relevant segment of your log file and put it here in your question so that it's easy to see. – roaima May 15 at 12:09
  • for example: Larry Heard ?? Where Life Begins (2003)[FLAC]{ML-9014}/03 Larry Heard - Insight.flac is actually Larry Heard — Where Life Begins (2003)[FLAC]{ML-9014}/03 Larry Heard - Insight.flac – Kusuma Nassim May 15 at 22:43
1

Ok I worked it out, it had nothing to do with utf-8, rather filenames triggering one of rsyncs three wildcard characters '*', '?', and '[' . The '?' and '*' weren't causing issues, however the '[' introduces a character class, such as [a-z] or [[:alpha:]], which caused mismatches.

After writing rsyncs progress to the log i added:

sed -i 's/\[/\*/g' /mnt/user/sync/music-manual/__sync-log

To replace '[' with '*'

is there a neat way of including this in the rsync command?

See INCLUDE/EXCLUDE PATTERN RULES

  • You can probably escape the [ to \[ with sed 's/\[/\\[/g', but you are right, I don't think there is any way to get rsync to interpret the patterns as literal strings. – meuh May 16 at 16:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.