Linux backup utility for incremental backups

Question

I'm looking for a backup utility with incremental backups, but in a more complicated way.

I tried rsync, but it doesn't seem to be able to do what I want, or more likely, I don't know how to make it do that.

So this is an example of what I want to achieve with it. I have the following files:

testdir
├── picture1
├── randomfile1
├── randomfile2
└── textfile1

I want to run the backup utility and basically create an archive (or a tarball) of all of these files in a different directory:

$ mystery-command testdir/ testbak
testbak
└── 2020-02-16--05-10-45--testdir.tar

Now, let's say the following day, I add a file, such that my structure looks like:

testdir
├── picture1
├── randomfile1
├── randomfile2
├── randomfile3
└── textfile1

Now when I run the mystery command, I will get another tarball for that day:

$ mystery-command testdir/ testbak
testbak
├── 2020-02-16--05-10-45--testdir.tar
└── 2020-02-17--03-24-16--testdir.tar

Here's the kicker: I want the backup utility to detect the fact that picture1, randomfile1, randomfile2 and textfile1 have not been changed since last backup, and only backup the new/changed files, which in this case is randomfile3, such that:

tester@raspberrypi:~ $ tar -tf testbak/2020-02-16--05-10-45--testdir.tar 
testdir/
testdir/randomfile1
testdir/textfile1
testdir/randomfile2
testdir/picture1
tester@raspberrypi:~ $ tar -tf testbak/2020-02-17--03-24-16--testdir.tar 
testdir/randomfile3

So as a last example, let's say the next day I changed textfile1, and added picture2 and picture3:

$ mystery-command testdir/ testbak
testbak/
├── 2020-02-16--05-10-45--testdir.tar
├── 2020-02-17--03-24-16--testdir.tar
└── 2020-02-18--01-54-41--testdir.tar
tester@raspberrypi:~ $ tar -tf testbak/2020-02-16--05-10-45--testdir.tar 
testdir/
testdir/randomfile1
testdir/textfile1
testdir/randomfile2
testdir/picture1
tester@raspberrypi:~ $ tar -tf testbak/2020-02-17--03-24-16--testdir.tar 
testdir/randomfile3
tester@raspberrypi:~ $ tar -tf testbak/2020-02-18--01-54-41--testdir.tar 
testdir/textfile1
testdir/picture2
testdir/picture3

With this system, I would save space by only backing up the incremental changes between each backup (with obviously the master backup that has all the initial files), and I would have backups of the incremental changes, so for example if I made a change on day 2, and changed the same thing again on day 3, I can still get the file with the change from day 2, but before the change from day 3.

I think it's kinda like how GitHub works :)

I know I could probably create a script that runs a diff and then selects the files to backup based on the result (or more efficiently, just get a checksum and compare), but I want to know if there's any utility that can do this a tad easier :)

Answer 1

I tried rsync, but it doesn't seem to be able to do what I want, or more likely, I don't know how to make it do that.

I know I could probably create a script that runs a diff and then selects the files to backup based on the result (or more efficiently, just get a checksum and compare), but I want to know if there's any utility that can do this a tad easier :)

rsync is precisely that program that copies based on a diff. By default, it copies only when there is a difference in last-modified time or size, but it can even compare by checksum with -c.

The trouble here is that you're tar'ing the backups. This becomes easier if you don't do that. I don't even know why you're doing it. It might make sense if you're compressing them, but you're not even doing that.

The Wikipedia article for Incremental Backups has an example rsync command that goes roughly:

rsync -va \
  --link-dest="$dst/2020-02-16--05-10-45--testdir/" \
  "$src/testdir/" \
  "$dst/2020-02-17--03-24-16--testdir/"

What it does is to hardlink files from the previous backup when they are unchanged from the source. There's also --copy-dest if you want it to copy instead (it's still faster when $dst is a remote or on a faster drive).

If you use a filesystem with subvolumes like btrfs, you can also just snapshot from the previous backup before rsync'ing. Snapshots are instantaneous and don't take additional space[1].

btrfs subvolume snapshot \
  "$dst/2020-02-16--05-10-45--testdir" \
  "$dst/2020-02-17--03-24-16--testdir"

Or if you're using a filesystem that supports reflinks, then you can also do that. Reflinks are done by making a new inode but referring to the same blocks as the source file, implementing COW support. It's still faster than regular copy because it doesn't read and write the data, and it also doesn't take additional space[1].

cp --reflink -av \
  "$dst/2020-02-16--05-10-45--testdir" \
  "$dst/2020-02-17--03-24-16--testdir"

Anyway, once having done something like that you can just do a regular rsync to copy the differences:

rsync -va \
  "$src/testdir/" \
  "$dst/2020-02-17--03-24-16--testdir/"

Though, you might want to add --delete, which would cause rsync to delete files from the destination that are no longer present in the source.

Another useful option is -i or --itemize-changes. It produces succinct, machine readable output that describes what changes rsync is doing. I normally add that option and pipe like:

rsync -Pai --delete \
  "$src/testdir/" \
  "$dst/2020-02-17--03-24-16--testdir/" \
|& tee -a "$dst/2020-02-17--03-24-16--testdir.log"

to keep record of the changes via easily grepable files. The |& is to pipe both stdout and stderr.

The -P is short for --partial and --progress. --partial keeps partially transferred files, but more importantly --progress reports per-file progress.

How this compares to archiving changes with tar

The above solutions result in directories that seem to hold everything. Even though that's the case, in total for any amount/frequency of backups, they would occupy around the same amount of space as having plain tar archives with only changes. That's because of how hardlinks, reflinks, and snapshots work. The use of bandwidth when creating the backups would also be the same.

The advantages are:

• backups are easy to restore with rsync and faster, since rsync would only transfer the differences from the backup.
• they're simpler to browse and modify if needed.
• file deletions can be encoded naturally as the file's absence in new backups. When using tar archives, one would have to resort to hacks, like to delete a file foo, mark it foo.DELETED or do something complicated. I've never used duplicity for example, but looking at its documentation, it seems it encodes deletions by adding an empty file of the same name in the new tar and holding the original signature of the file in a separate .sigtar file. I imagine it compares the original signature with that of an empty file to differentiate between a file deletion and a change to an actual empty file.

If one still wants to setup each backup as only holding the files that are different (added or modified), then one can use the --link-dest solution described above and then delete the hardlinks using something like the following:

find $new_backup -type f ! -links 1 -delete

[1] Strictly speaking, they do use additional space in the form of duplicate metadata, like the filename and such. However, I think anyone would consider that insignificant.

Answer 2

Although tar does have an incremental mode there are a couple of more comprehensive tools to do the job:

• Duplicity
• Duplicati

They not only support incremental backups, it's easy to configure a schedule on which a full backup needs to be taken. For example in duplicity: duplicity --full-if-older-than 1M will make sure a full backup has run. They also support going back in time to a specific file, with plain tar you'll have to go through all incremental files till you found one which contains the right file.

Additionally they do support encryption and uploading to a variety of backends (like sftp, blob storage, etc). Obviously if you encrypt, don't forget to make a good backup of your keys to a secondary backup!

Another important aspect is that you can verify the integrity of your backups, ensuring you can restore, eg using duplicity verify.

I would negatively advise on a git based backup strategy. Large restores take significant time.

Answer 3

And why are you not considering git itself?

The strategy you describe, after one full and two incremental backups, has it's complications when you continue. It is easy to make mistakes, and it can get very inefficient, depending on the changes. There would have to be a kind of rotation, ie from time to time you make a new full backup - and then do you want to keep the old one or not?

---

Given a working dir "testdir" containing some project (files, and subdirs), git makes by default a hidden .git subdir for the data. That would be for the local, additional version control features. For backup, you can archive/copy it away to a medium or clone it via network.

The revision control you get (without asking for) is a side effect of git's differential storage.

You can leave out all the forking/branching and so on. This means you have one branch called "master".

Before you can commit (actually write to the git archive/repo), you have to configure a minimal user for the config file. Then you should first learn and test in a subdir (maybe tmpfs). Git is just as tricky as tar, sometimes.

Anyway, as a comment says: backing up is easy, hard part is the restoring.

---

Disadvantages of git would be just the small overhead/overkill.

Advantages are: git tracks content and file names. It only saves what is necessary, based on a diff (for text files at least).

---

Example

I have 3 files in a dir. After git init, git add . and git commit I have a 260K .git dir.

Then I cp -r .git /tmp/abpic.git (a good place to save a backup:). I rm the 154K jpg, and also change one text file. I also rm -r .git.

  ]# ls
    atext  btext

  ]# git --git-dir=/tmp/abpic.git/ ls-files
    atext
    btext
    pic154k.jpg

Before restoring the files I can get the precise differences:

]# git --git-dir=/tmp/abpic.git/ status
On branch master
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   atext
        deleted:    pic154k.jpg

no changes added to commit (use "git add" and/or "git commit -a")

Here I want to follow the git restore hint.

After git --git-dir=/tmp/abpic.git/ restore \*:

]# ls -st
total 164
  4 atext  156 pic154k.jpg    4 btext

The jpeg is back, and text file btext has not been updated (keeps timestamp). The modifications in atext are overwritten.

To reunite the repo and the (working) dir you can just copy it back.

]# cp -r /tmp/abpic.git/ .git
]# git status
On branch master
nothing to commit, working tree clean

The files in the current dir are identical to the .git archive (after the restore). New changes will be displayed and can be added and committed, without any planning. You only have to store it to another medium, for backup purposes.

---

After a file is modified, you can use status or diff:

]# echo more >>btext 

]# git status
On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   btext

no changes added to commit (use "git add" and/or "git commit -a")

]# git diff
diff --git a/btext b/btext
index 96b5d76..a4a6c5b 100644
--- a/btext
+++ b/btext
@@ -1,2 +1,3 @@
 This is file b
 second line
+more
#]

And just like git knows about "+more" in file 'btext', it will also only store that line incrementally.

After git add . (or git add btext) the status command switches from red to green and the commit gives you the info.

]# git add .
]# git status
On branch master
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        modified:   btext

]# git commit -m 'btext: more'
[master fad0453] btext: more
 1 file changed, 1 insertion(+)

And you can really get at the contents, somehow:

]# git ls-tree @
100644 blob 321e55a5dc61e25fe34e7c79f388101bd1ae4bbf    atext
100644 blob a4a6c5bd3359d84705e5fd01884caa8abd1736d0    btext
100644 blob 2d550ffe96aa4347e465109831ac52b7897b9f0d    pic154k.jpg

And then the first 4 hex hash digits

]# git cat-file blob a4a6
This is file b
second line
more

---

To travel back in time by one commit it is:

]# git ls-tree @^
100644 blob 321e55a5dc61e25fe34e7c79f388101bd1ae4bbf    atext
100644 blob 96b5d76c5ee3ccb7e02be421e21c4fb8b96ca2f0    btext
100644 blob 2d550ffe96aa4347e465109831ac52b7897b9f0d    pic154k.jpg

]# git cat-file blob 96b5
This is file b
second line

btext's blob has a different hash before the last commit, the others have the same.

An overview would be:

]# git log
commit fad04538f7f8ddae1f630b648d1fe85c1fafa1b4 (HEAD -> master)
Author: Your Name <[email protected]>
Date:   Sun Feb 16 10:51:51 2020 +0000

    btext: more

commit 0bfc1837e20988f1b80f8b7070c5cdd2de346dc7
Author: Your Name <[email protected]>
Date:   Sun Feb 16 08:45:16 2020 +0000

    added 3 files with 'add .'

Instead of manually timestamped tar files you have commits with a message and date (and an author). Logically attached to these commits are the file lists and contents.

Simple git is 20% more complicated than tar, but you get decisive 50% more functionality from it.

---

I wanted to make OP's third change: change a file plus two new 'picture' files. I did, but now I have:

]# git log
commit deca7be7de8571a222d9fb9c0d1287e1d4d3160c (HEAD -> master)
Author: Your Name <[email protected]>
Date:   Sun Feb 16 17:56:18 2020 +0000

    didn't add the pics before :(

commit b0355a07476c8d8103ce937ddc372575f0fb8ebf
Author: Your Name <[email protected]>
Date:   Sun Feb 16 17:54:03 2020 +0000

    Two new picture files
    Had to change btext...

commit fad04538f7f8ddae1f630b648d1fe85c1fafa1b4
Author: Your Name <[email protected]>
Date:   Sun Feb 16 10:51:51 2020 +0000

    btext: more

commit 0bfc1837e20988f1b80f8b7070c5cdd2de346dc7
Author: Your Name <[email protected]>
Date:   Sun Feb 16 08:45:16 2020 +0000

    added 3 files with 'add .'
]# 

So what did that Your Name Guy do exactly, in his two commits, shortly before 6 pm?

The last commit's details are:

]# git show
commit deca7be7de8571a222d9fb9c0d1287e1d4d3160c (HEAD -> master)
Author: Your Name <[email protected]>
Date:   Sun Feb 16 17:56:18 2020 +0000

    didn't add the pics before :(

diff --git a/picture2 b/picture2
new file mode 100644
index 0000000..d00491f
--- /dev/null
+++ b/picture2
@@ -0,0 +1 @@
+1
diff --git a/picture3 b/picture3
new file mode 100644
index 0000000..0cfbf08
--- /dev/null
+++ b/picture3
@@ -0,0 +1 @@
+2
]# 

And to check the second-to-last commit, whose message announces two pictures:

]# git show @^
commit b0355a07476c8d8103ce937ddc372575f0fb8ebf
Author: Your Name <[email protected]>
Date:   Sun Feb 16 17:54:03 2020 +0000

    Two new picture files
    Had to change btext...

diff --git a/btext b/btext
index a4a6c5b..de7291e 100644
--- a/btext
+++ b/btext
@@ -1,3 +1 @@
-This is file b
-second line
-more
+Completely changed file b
]# 

This happened because I tried git commit -a to shortcut git add ., and the two files were new (untracked). It showed in red with git status, but as I say git is not less tricky than tar, or unix.

---

"Your debutante just knows what you need, but I know what you want" (or the other way round. Point is it's not always the same)

Answer 4

I would recommend you get a look at Borg Backup.

This will handle backups that:

• Are deduplicated. This indirectly makes it differential backups, but has more advantages:

  • It will handle multiple copies of the same file
  • Or even of the same blocks within different files
  • Will help with files that grow (like logs)
  • Will help with files that are renamed (like logs in some rotation setups)

• Are compressed

• Can be mounted like a regular remote file system (you can mount any of the previous backups)

It will manage pruning of old backups using rules such as "keep one daily backup for a week, one weekly backup for a month, one monthly backup for a year"

It's really easy to set up and use.

Answer 5

Update:

Please see some caveats here: Is it possible to use tar for full system backups?

According to that answer, restoration of incremental backups with tar is prone to errors and should be avoided. Do not use the below method unless you're absolutely sure you can recover your data when you need it.

---

According to the documentation you can use the -g/--listed-incremental option to create incremental tar files, eg.

tar -cg data.inc -f DATE-data.tar /path/to/data

Then next time do something like

tar -cg data.inc -f NEWDATE-data.tar /path/to/data

Where data.inc is your incremental metadata, and DATE-data.tar are your incremental archives.

Answer 6

I recommend star for incremental backups, since star has been verified to reliably support incremental dumps and restores. The latter is what does not work with GNU tar when you rename directories even though it is advertized since 28 years.

Please read the star man page at http://schilytools.sourceforge.net/man/man1/star.1.html

The section about incremental backups is currently starting at page 53.

To download the source, get the schilytools tarball from http://sourceforge.net/projects/schilytools/files/

Check Is it possible to use tar for full system backups? for a verification of the GNU tar bug.

Answer 7

Take a look at restic. It does incremental back ups using an algorithm called deduplication. It's also very easy to use so its great for a beginner or advanced command line user.

Answer 8

You could try BackupPC.

It allows for incremental backups, you can decide how often to do them, how many to keep and when you look at them you can see them consolidated or just the actual incremental backup. It also dedups complete files, should they be present in different backups of the same or different hosts.

It's most likely already packaged for your distribution.

Answer 9

This is not exactly what you are requesting, because it doesn't use tar. But it does use rsync, and it has worked very well for me. On of the abilities that I really like is the ability to drop incremental restore points over time without loosing points before or after the one I am dropping. This allows me to, for example, have daily backups for the last 2 weeks, then thin those out once they get 2 weeks old so they are weekly for a couple months, then further thin those out until they are monthly for a quarter or two, then thin those out to about quarterly over the timespan of years. I have a python script that I can share that can prune these automatically if you want. (Obviously, it comes with NO WARRANTY as letting a computer automatically delete backups sounds a bit scary.)

What I do is use a ZFS pool & filesystem for storing backups. With the ZFS filesystem, which is (thankfully!) now usable on linux, you can take snapshots. When you write to a filesystem that has been snapshotted, it (smartly) writes a new version of only the changed blocks, thus making it an incremental backup. Even easier is that all of your snapshots can be mounted as a full (read only) Unix filesystem, that you can use all of your normal tools on to look at and copy from. Want to see what that file looked like 2 months ago? Just cd to the right folder and use less or vim or whatever to look at it. Want to see when a (hacked) wordpress install you were backing up went off the rails? Just do a grep for an identifying mark with something like grep -in /zfsbackup/computername/.zfs/snapshots/*/var/www/html/wp-config.php" "somebadstring"

You can even use Linux's LUKS system to do disk encryption and then present the mapped device as "drives" to ZFS, giving you an encrypted backup.

If you ever need to migrate your backups to a new drive, you can use zfs send & receive to move the entire filesystem.

It has been a year or two since I've set it up (I just keep adding on incremental backups and haven't needed to upgrade my backup drive for a while), so these will be rough instructions. Bear with me, or better yet, edit them.

First, make sure you have zfs, rsync, and, if you want to encrypt your backups, the LUKS tools installed.

First, create any partition layout you might want on your backup drive. (You may want to make a small unencrypted partition that has scripts for running the backup.)

Then, if you want disk encryption, encrypt the partition with LUKS (example assumes a backup drive of /dev/sde and a partition /dev/sde2 since /dev/sde1 is probably scripts):

sudo cryptsetup luksFormat /dev/sde2

(Put in a nice strong passphrase).

If you are doing disk encryption, now you need to open the volume:

sudo cryptsetup luksOpen /dev/sde2 zfsbackuppart1

(Now an unencrypted version of the raw device should be available (mapped) at /dev/mapper/zfsbackuppart1).

Now, create you ZFS pool (group of drive(s) holding data, multiple drives/devices can be used for RAID if you wish):

sudo zpool create zfsbackup /dev/mapper/zfsbackuppart1

This will create a ZFS pool named "zfsbackup".

Now, create a filesystem for each machine you are backing up:

sudo zfs create zfsbackup/machinename

And create a folder for each partition you want to back up from the source machine:

sudo mkdir /zfsbackup/machinename/slash/
sudo mkdir /zfsbackup/machinename/boot/

Then, use rsync to copy files to there:

sudo rsync -avx --numeric-ids --exclude .gvfs / /zfsbackup/machinename/slash/ --delete-after
sudo rsync -avx --numeric-ids --exclude .gvfs /boot/ /zfsbackup/machinename/boot/ --delete-after

Take a snapshot:

zfs snapshot zfsbackup/machinename@`date +%F_%T`

To disconnect the drive when you are done:

zpool export zfsbackup
# Next line, for each underlying encrypted block device, if using encryption:
cryptsetup luksClose zfsbackuppart1

And to set it up when taking another backup in the future, before the above rsync commands:

cryptsetup luksOpen /dev/sde2 zfsbackuppart1
zpool import zfsbackup

Let me know if you need more info on this approach, or are interested in a script to thin out backups as they get farther back in time.

And, yes, you can backup a whole system this way -- you just have to create partitions/filesystems (which don't have to match the original layout -- a great way to migrate stuff!), tweak /etc/fstab, and install GRUB & have it rescan/rebuild the GRUB config.

Answer 10

One possibility is AMANDA, the Advanced Maryland Automatic Network Disk Archiver, which among loads of other features, also support incremental backups.

Answer 11

The very old BSD rdump command would do backup levels. Level 0 backed the entire file system. Level 1 would backup what changed since the zero level, Level 2 would do all backups since level 1. To really the files required zero to be applied to the filesystem, then level 2, followed by 3 (if used). Rdump has the misfortune of writing to a tape...

The following shell script will do what you want using tar, find and egrep. It used the -newer flag of find comparing "newer" against a "touch"ed that was created when the script was last run. The script was last run on a Solaris 4.3bsd from the 1990's to copy source files between a UNIX development machine and laptops using kermit.

The script is rather fragile because if the script "touch"es the update time file, and the tar fails, then the time file has to be recreated. The tar file has the name of the date+hour+minute+second. ` #!/bin/sh -vx

echo finding newer files than update ...

cd /home/programs/gis

HOME_LOC=/home/programs/gis/
TAR_FILE=${HOME_LOC}stage/u`date +%m%d%H%M`.tar.bz2

#run find one using the -o (or) syntax
(
find . -newer ${HOME_LOC}/update  \
      \(     -name '*.[hcsf]' -o -name '*.asm' \
       -o -name '*.ma*'    -o -name '*.cpp' \
       -o -name '*.msg'    -o -name '*.hpp' \
       -o -name 'Makefile' -o -name '*.inl'                 \
       -o -name 'grid*.*'   -o -name '*.glb' \)  \
        -print ) | \
       egrep -v 'SCCS|stage|local.h' | tee /usr/tmp/x.$$

echo copying files:

tar -cjhf ${TAR_FILE} `cat /usr/tmp/x.$$`

# rm  /usr/tmp/x.$$

if test -r ${TAR_FILE} 
then
    touch ${HOME_LOC}/update
fi

`

Answer 12

It looks like you are trying to do differential backups. I've used 7zip (as I also had to compress and encrypt) in the past for this. There are many tutorials available, for example this one.

Answer 13

Look for "Mike's handy rotating file system snapshot utility". It's basically just a convenience script that wraps around rsync to make rotating incremental backups of a directory. It will minimize memory usage by linking back to earlier incremental backups for all files that did not change.

See Easy Automated Snapshot-Style Backups with Linux and Rsync for a full introduction along with an explanation for how it works.

Answer 14

Consider using bup:

• No need to worry about the distinction of full/incremental/decremental backups.
• Backups are as fast as incremental backups
• Restores are as simple as from full backups
• Based on the storage format of git, extended to support really large backups
• Complete deduplication across files (and parts of files), versions, and branches. This is particularly beneficial if you need to backup more than one machine or several VM snapshots.
• Can add error-correction codes to the archive using par2
• Easy access to the entire backup tree, either with direct commands or by mounting the tree as a FUSE filesystem

Caveats I have found so far:

• Still young and in alpha stage (current version is 0.x)
• Needs Python 2, so it's not exactly light in terms of dependencies
• bup uses file-locking in the archive folder. This means that the filesystem hosting the archive has to support file-locking. This might be a problem with SMB shares. (NFS works however.)
• Backups to or from remote hosts (via ssh) require bup to be installed there.
• For your backup names (which are Git branches) you might be tempted to use something containing slashes (/). That is not a problem for git, but bup ls gets confused by those, so do not do that. (You can change the branch name with a suitable git branch -m command if you get bitten by that.)

Answer 15

In addition to all the excellent solutions provided above, I have one more approach. Since you have not mentioned the filesystem that you are using, I am assuming that you are not using zfs or btrfs. These filesystems have built-in capabilities to do snapshots that are not copies of the entire filesystem, but just the incrementals. A typical approach would be this:

1. Create a zfs/btrfs filesystem.
2. Locate all your data (that needs to be tracked) on this filesystem.
3. Set up snapshots
4. Set up snapshot retention/rotation (else your snapshots will keep growing and occupy disk space).

For example, if you set up a retention of 5 days, you can go back up to 5 days in time.