5

I use rsync to make backups:

rsync -a --link-dest=PATHTO/$PREVIOUSBACKUP $SOURCE $CURRENTBACKUP

This way I save space due to using hard links.

The problem appears when I need to backup a huge file which is always changing (virtual machine image).

Is it possible to hardlink not whole the image, but only it's changed parts? Is there any kind of tool that can take care of this?

  • 3
    The key that may help finding an answer is "incremental backup" or "delta backup". – Lekensteyn Mar 27 '14 at 13:25
  • 1
    Yes, in your case I'd go with a different backup system that can do incrementals / differentials. E.g., bacula. – Mark Wagner Mar 28 '14 at 16:50
3

There are a number of things that could be done here. Note that none of them actually use hard links since they can only point to a full file. Using the btrfs filesystem opens up some very useful possibilities here. Note that btrfs is currently (most recent version is v3.13) still experimental. However, its COW (copy-on-write) ability is perfect for this kind of thing (provided of course that it is acceptable to have the backup on the same filesystem). Consider a btrfs filesystem mounted on /mnt, you can make an atomic snapshot of the whole filesystem with:

btrfs subvolume snapshot /mnt /mnt/snapshot

To allow for partial snapshots, you have to put your files to be backed up inside a subvolume rather than a directory. Eg:

btrfs subvolume create /mnt/subvol
mv stuff /mnt/subvol
btrfs subvolume snapshot /mnt/subvol /mnt/subvol_snapshot

Aside from using btfrs, you could also consider mounting the virtual machine image on one or both sides of the backup and using rsync between the two mount points.

This blog shows how to mount a Virtual Box .vdi image using qemu-utils. The commands as root (untested):

modprobe nbd
qemu-nbd -c /dev/nbd0 <vdi-file>
mount /dev/nbd0p1 /mnt
...
umount /mnt
qemu-nbd -d /dev/nbd0

Finally, the simplest approach which may be of some use is the --inplace option for rsync. From the man page:

--inplace
          This option changes how rsync transfers a file when its data needs to
          be updated: instead of the default method of creating a new copy of the
          file and moving it into place when it is complete, rsync instead writes
          the updated data directly to the destination file.
...
          This option is useful for transferring large files with block-based
          changes or appended data, and also on systems that are disk bound, not
          network bound.

The problem here of course is that there isn't any benefit to using this in combination with --link-dest (in rsync versions <2.6.4 the two are incompatible altogether) as a copy of the file will still have to be created at the destination.

  • 1
    btrfs snapshotting is not a backup, if the underlying storage breaks, then your data is still (partially) lost. The --inplace option is unfortunately not going to help either, the OPs usage concerns using rsync as a snapshotting tool. Nevertheless, your suggestion of mounting a image is still helpful (+1). – Lekensteyn Mar 27 '14 at 13:24
  • @Lekensteyn, updated, I forgot about the --link-dest. The btrfs snapshot is still effectively a backup, but, yes, it doesn't protect against the same range of data loss cases as backing up to a different filesystem/location. Note, however, that btrfs can span multiple disks. – Graeme Mar 27 '14 at 13:51
  • 1
    You can use btrfs-send to send your snapshots to another btrfs volume (or just save the stream as a file anywhere else) to make off-site incremental backups. – Vladimir Panteleev Feb 28 '16 at 6:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.