In rsync commands, is it possible to configure the "fuzzy-basis-destination-file-selection algorithm"?
I use rsync's --fuzzy argument successfully most of the time when transferring local Proxmox backups offsite, but sometimes I can see (in my verbose logging) that rsync chooses the wrong existing-destination-file for its "fuzzy basis".
This is especially wasteful when it does this mistake on a 200GB file, because it causes a lot of data to transfer that wouldn't have needed transferring, had the correct "fuzzy basis" been chosen.
The tricky part about my transfer is that, although the source-folder and destination-folder are the same each time, all the file-names in the source-folder are different each time I transfer them. Yet, the "contents of each renamed file" is 95% identical to the previously named file (that the destination still has from the previous transfer).
A simplified example of how the names change is below.
# Source Folder's new files (from today's on-site backup):
file100-2019_09-01_12am.log
file100-2019_09-01_12am.lzo
file101-2019_09-01_12am.log
file101-2019_09-01_12am.lzo
file102-2019_09-01_12am.log
file102-2019_09-01_12am.lzo
# Destination-Folder's old files (from yesterday's off-site backup):
file100-2019_08-31_12am.log
file100-2019_08-31_12am.lzo
file101-2019_08-31_12am.log
file101-2019_08-31_12am.lzo
file102-2019_08-31_12am.log
file102-2019_08-31_12am.lzo
Most of the time rsync chooses the correct "fuzzy basis", but sometimes it chooses the wrong one. Here's a log entry of the wrong one being chosen:
fuzzy basis selected for file100-2019_08-31_12am.lzo:
file101-2019_09-01_12am.lzo
As you can see, rsync chose file101...lzo as the "fuzzy basis" for file100...lzo. However, those two files do not have much in common (with respect to content) and therefore a great deal of data transfers due to this wrong "fuzzy basis" selection. The correct "fuzzy basis" for file file100-2019_09-01_12am.lzo is file100-2019_08-31_12am.lzo. So why did it choose file101-2019_08-31_12am.lzo instead? How could I have optimized the "fuzzy basis selection" to have chosen file100-2019_08-31_12am.lzo instead?
In my case, the fuzzy-basis-selection-algorithm needs to select the existing destination-file that:
- Has the same file extension as the source file
- Begins with the most consecutively identical characters as the source file
In rsync commands, is it possible to configure the "fuzzy-basis-destination-file-selection algorithm"?
