2

In rsync commands, is it possible to configure the "fuzzy-basis-destination-file-selection algorithm"?

I use rsync's --fuzzy argument successfully most of the time when transferring local Proxmox backups offsite, but sometimes I can see (in my verbose logging) that rsync chooses the wrong existing-destination-file for its "fuzzy basis".

This is especially wasteful when it does this mistake on a 200GB file, because it causes a lot of data to transfer that wouldn't have needed transferring, had the correct "fuzzy basis" been chosen.

The tricky part about my transfer is that, although the source-folder and destination-folder are the same each time, all the file-names in the source-folder are different each time I transfer them. Yet, the "contents of each renamed file" is 95% identical to the previously named file (that the destination still has from the previous transfer).

A simplified example of how the names change is below.

# Source Folder's new files (from today's on-site backup):
file100-2019_09-01_12am.log
file100-2019_09-01_12am.lzo
file101-2019_09-01_12am.log
file101-2019_09-01_12am.lzo
file102-2019_09-01_12am.log
file102-2019_09-01_12am.lzo

# Destination-Folder's old files (from yesterday's off-site backup):
file100-2019_08-31_12am.log
file100-2019_08-31_12am.lzo
file101-2019_08-31_12am.log
file101-2019_08-31_12am.lzo
file102-2019_08-31_12am.log
file102-2019_08-31_12am.lzo

Most of the time rsync chooses the correct "fuzzy basis", but sometimes it chooses the wrong one. Here's a log entry of the wrong one being chosen:

fuzzy basis selected for file100-2019_08-31_12am.lzo:
file101-2019_09-01_12am.lzo

As you can see, rsync chose file101...lzo as the "fuzzy basis" for file100...lzo. However, those two files do not have much in common (with respect to content) and therefore a great deal of data transfers due to this wrong "fuzzy basis" selection. The correct "fuzzy basis" for file file100-2019_09-01_12am.lzo is file100-2019_08-31_12am.lzo. So why did it choose file101-2019_08-31_12am.lzo instead? How could I have optimized the "fuzzy basis selection" to have chosen file100-2019_08-31_12am.lzo instead?

In my case, the fuzzy-basis-selection-algorithm needs to select the existing destination-file that:

  1. Has the same file extension as the source file
  2. Begins with the most consecutively identical characters as the source file

In rsync commands, is it possible to configure the "fuzzy-basis-destination-file-selection algorithm"?

  • Given that no one has answered or even commented on my question, I'm assuming that this isn't possible yet. So, I went ahead an filed a feature request. – Lonniebiz Sep 1 '19 at 23:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.