Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
Is there a backup tool which is "intelligent" enough to notice that a folder or large files may have been renamed between two backups? Maybe even if their location changed (not too complicated)?
Is it clear what I try to ask for?
My backup methods for now have all added the new dirs to the existing backup. How to "copy the difference" without touching the unchanged parts?
Yes, deduplicating backup tools like restic and borgbackup would do this.
These would detect that a given chunk of data (not necessarily a whole file) was already present in the older backup and would not store it again. It would also detect the same chunk in other files, so your fifteen copies of the same MP3 file would only be stored once.
I use it on a machine where I have two sets of JPEG files, one in macOS's "Photos" album archive and then the same photos as originals in a structured directory hierarchy based on dates. This is 2 * 60 Gb of data, but restic only stores 60 Gb since it's deduplicating it.
Another example is another machine (OpenBSD this time), where I have two or three different checkouts of the same Git repository (don't ask why). These, too, are deduplicated to the degree possible and will only use approximately the size taken by the files that are different (the 270 Mb .git directory is mostly the same and will be stored only once in the backup).
Moving a directory would likewise only ever result in a few kilobytes or so of data being written to the backup (depending on the size of the directory structure). I renamed one of these 270 Mb Git repositories as a test and ran a backup. This wrote just over 500 Kb to the backup (this data would be information about the locations of files and their metadata such as ownership and timestamps etc.)
A deduplicating backup tool would also allow you to backup data from multiple machines to the same location and have that data be deduplicated across the machines so that, for example, your Dropbox folder on three machines does not get stored three times (this is at least possible with restic).
The downside of using a deduplicating backup tool is that you can't browse the backups as files (borgbackup may allow mounting a snapshot as a directory somehow, but I haven't investigated it because it uses Fuse, which is unsupported by OpenBSD). One would have to use the backup tool to restore a snapshot or the wanted files from a snapshot.
I'm using restic because that allows me to back up over SFTP to a server where restic itself is not installed. Another way of doing this would be to run a restic REST server using rclone (rclone server restic ...) on the backup server and let the restic clients talk to that.
borgbackup allows for compression of the data chunks, but I think it requires that borgbackup is installed on the machine where the backups live. borgbackup is also (IMHO) slightly harder to configure.
The most recent versions of restic also support compression (using zstd).
detect-renamedpatch.