Yes, deduplicating backup tools like restic
and borgbackup
would do this.
These would detect that a given chunk of data (not necessarily a whole file) was already present in the older backup and would not store it again. It would also detect the same chunk in other files, so your fifteen copies of the same MP3 file would only be stored once.
I use it on a machine where I have two sets of JPEG files, one in macOS's "Photos" album archive and then the same photos as originals in a structured directory hierarchy based on dates. This is 2 * 60 Gb of data, but restic
only stores 60 Gb since it's deduplicating it.
Another example is another machine (OpenBSD this time), where I have two or three different checkouts of the same Git repository (don't ask why). These, too, are deduplicated to the degree possible and will only use approximately the size taken by the files that are different (the 270 Mb .git
directory is mostly the same and will be stored only once in the backup).
Moving a directory would likewise only ever result in a few kilobytes or so of data being written to the backup (depending on the size of the directory structure). I renamed one of these 270 Mb Git repositories as a test and ran a backup. This wrote just over 500 Kb to the backup (this data would be information about the locations of files and their metadata such as ownership and timestamps etc.)
A deduplicating backup tool would also allow you to backup data from multiple machines to the same location and have that data be deduplicated across the machines so that, for example, your Dropbox folder on three machines does not get stored three times (this is at least possible with restic
).
The downside of using a deduplicating backup tool is that you can't browse the backups as files (borgbackup
may allow mounting a snapshot as a directory somehow, but I haven't investigated it because it uses Fuse, which is unsupported by OpenBSD). One would have to use the backup tool to restore a snapshot or the wanted files from a snapshot.
I'm using restic
because that allows me to back up over SFTP to a server where restic
itself is not installed. Another way of doing this would be to run a restic
REST server using rclone
(rclone server restic ...
) on the backup server and let the restic
clients talk to that.
borgbackup
allows for compression of the data chunks, but I think it requires that borgbackup
is installed on the machine where the backups live. borgbackup
is also (IMHO) slightly harder to configure.
The most recent versions of restic
also support compression (using zstd
).
detect-renamed
patch.