Is there a backup tool which is "intelligent" enough to notice that a folder or large files may have been renamed between two backups? Maybe even if their location changed (not too complicated)?

Is it clear what I try to ask for?

My backup methods for now have all added the new dirs to the existing backup. How to "copy the difference" without touching the unchanged parts?

  • Use btrfs snapshots and send only the difference, or use rsync with detect-renamed patch. – Ipor Sircer Nov 18 '18 at 19:40

Yes, deduplicating backup tools like restic and borgbackup would do this.

These would detect that a given chunk of data (not necessarily a whole file) was already present in the older backup and would not store it again. It would also detect the same chunk in other files, so your fifteen copies of one and the same MP3 file would only be stored once.

I use it on a machine where I have two sets of JPEG files, one in macOS's "Photos" album archive, and then the same photos as originals in structured directory hierarchy based on dates. This is 2 * 60 Gb of data, but restic only stores 60 Gb since it's deduplicating it.

Another example is another machine (OpenBSD this time), where I have two or three different checkouts of the same Git repository (don't ask why). These too are deduplicated to the degree possible and will only use approximately the size taken by the files that are actually different (the 270 Mb .git directory is mostly the same and will be stored only once in the backup).

Moving a directory would likewise only ever result in a few kilobytes or so of data being written to the backup (depending on the size of the directory structure). As a test, I renamed one of these 270 Mb Git repositories and ran a backup. This wrote just over 500 Kb to the backup (this data would be information about the locations of files and their metadata such as ownership and timestamps etc.)

A deduplicating backup tool would also allow you to backup data from multiple machines to the same location and have that data be deduplicated across the machines, so that, for example, your Dropbox folder on three machines does not get stored three times (this is at least possible with restic).

The downside of using a deduplicating backup tool is that you can't browse the backups as files (borgbackup may allow mounting a snapshot as a directory somehow, but I haven't really investigated it because it uses Fuse which is unsupported by OpenBSD). One would have to use the backup tool to restore a snapshot or the wanted files from a snapshot.

I'm using restic because that allows me to backup over SFTP to a server where restic itself is not installed.

borgbackup allows for compression of the data chunks, but I think it requires that borgbackup is installed on the machine where the backups live. borgbackup is also (IMHO) slightly harder to configure.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.