7

I have a few Ubuntu (and potentially RHEL) servers I'd like to back up to a central backup server (via rsh/SSH). I'd like to do both snapshot and incremental backups. The goal is that if one of the backed-up servers were to suddenly die, I could install a bootstrap image, restore from the backup and ideally be up-and-running immediately thereafter.

These servers have unique users and groups and some different gids and uids for the same names. I was considering rsync (local backups work fine, and I have fine-grained control so I can back up xattrs and ACLs as well) but then I realized rsync to a remote server likely won't retain these nuances as expected.

So, what's a better way to accomplish this? Is there a package that is commonly used for this purpose?

Improve this question
4
  • 1
    Sounds like you're describing Bacula, or similar.
    – derobert
    Oct 25, 2012 at 5:37
  • Possibly so... I'll look into that.
    – MartyMacGyver
    Oct 25, 2012 at 6:03
  • Instead of posting your answer as an edit to the question, click 'answer your own question' and post it as an answer... or, even better, edit it into Stephane Chazelas's answer.
    – derobert
    Oct 26, 2012 at 16:19
  • Added my own answer instead. rsync - which I started with and which I was aware of most of the options for - isn't the only solution, but it'll do for the time being.
    – MartyMacGyver
    Oct 26, 2012 at 17:05

3 Answers 3

Reset to default
4

Yes rsync has options to preserve ACLs, extended attribute, ext2 attributes, hard links, numerical userids, sparseness...

I used to have a system that rsynced filesystems onto a backup server with a btrfs file system (to do snapshots on the backup server), and due to stability issues, moved to zfs-on-linux (which unfortunately doesn't support POSIX ACLs yet, but there are work arounds).

rsync options you may be interested in: --archive --one-file-system --xattrs --hard-links --delete --numeric-ids --sparse --acls and potentially those for in-place syncing.

One thing rsync doesn't do is detect renames (though there is a patch for that, but I'm not sure how much it can be trusted, and it only detects file renames).

zfs has zfs send to send only the modifications done since a given snapshot (to another zfs zpool using zfs receive). If I understand correctly, that feature has been added very recently to btrfs as well. So if backing up a btrfs FS onto another btrfs FS on the latest kernel, that could be an option.

Improve this answer
4
  • I stated some of the rsync options myself in the original question, having used them during local backups (to another partition/disk on the same system). But are UIDs, GIDs, ACLs and so on preserved when backing up via SSH to another box entirely, one that may not have the groups or users the original machine does? Architecturally the answer seems to be "no" so I asked, what's a better option (one that will work on more than just Ubuntu, and with ext3, ext4, xfs, etc.)?
    – MartyMacGyver
    Oct 25, 2012 at 9:10
  • A filesystem doesn't care about the system's user database. All it cares about is user ids. So with --numeric-id, there is no problem sending data to another machine, it doesn't matter whether there are users with those uids or not. Same for ACLs (which are actually stored as extended attributes). The only issues that you may find is when the target FS doesn't support the same feature set as the source one (like POSIX ACLs for ZFS)
    – Stéphane Chazelas
    Oct 25, 2012 at 9:48
  • You can also try the --fake-super option on the remote rsync. Example from the manpage: rsync -av --rsync-path="rsync --fake-super" /src/ host:/dest/
    – derobert
    Oct 25, 2012 at 14:36
  • I'll look into this option, and will run some tests to see what happens on the target (and during a restore).
    – MartyMacGyver
    Oct 25, 2012 at 17:40
4

Have a look at rsnapshot, which is a perl wrapper around the rsync command-line utility. Rsnapshot adds an extra config layer to rsync that makes incremental backups easy. (Older versions of a file are hard linked to the current version when they're unchanged. So each incremental version of your backed-up filesystem looks like a full copy.)

Another similar possibility is duplicity, which instead is based on librsync. I understand this has a more polished UI, but I haven't used it. Or rdiff-backup, which I think is from the same developer, or at any rate that was once true.

There are of course lots and lots of backup systems. Here's a list of ones available on FreeBSD, most of which will also be available on Linux. But I think the three I mentioned above are natural ones to consider if you're used to rsync. (A wonderful tool.)

Improve this answer
5
  • Will these retain the proper metadata, GIDs, UIDs and so on, even if the target host is unaware of those groups and users?
    – MartyMacGyver
    Oct 25, 2012 at 17:39
  • I know that rsync will, with the right command-line arguments, and that rsnapshot can be configured to pass any custom arguments to rsync that you want. (If it's not already correctly configured out of the box.) I'd use the arguments rsync -a --delete (these will certainly be there by default) plus perhaps --numeric-ids, perhaps --hard-links, perhaps --xattrs or --acls if you need to retain those. Highly recommend familiarizing yourself with man rsync.
    – dubiousjim
    Oct 25, 2012 at 18:32
  • I had these and more already, but what I didn't know was whether this would work well or poorly for this purpose. Turns out they have been working OK (though it'd be nice to be able to archive this entirely with such options, instead of having it live as a flat filesystem with numeric IDs and such... unfortunately tar AFAIK isn't up to the task).
    – MartyMacGyver
    Oct 27, 2012 at 9:11
  • I'm not sure what you mean here by "flat". As to having the backup exposed as a live filesystem, iirc rdiff-backup doesn't do that and probably duplicity doesn't either since it uses a similar underlying mechanism.
    – dubiousjim
    Oct 28, 2012 at 8:42
  • Flat as in my backup is a bunch of files and folders on the disk, versus blob(s) that contain the data much like a tarball might.
    – MartyMacGyver
    Oct 28, 2012 at 10:23
3

Edit: Thanks all! I'm now aware of some other packages to do the job, and I know a lot more about rsync internals. For those who may also wonder this, if you rsync to another machine with the --numeric_ids option, it won't try and map or create the UIDs/GIDs, it'll just pass them as-is. The ownership of the items in the backup directory may look odd, but it'll match 1:1 on restore (assuming part of what you restore involves the original mapping files, if you're restoring to a new machine).

Here's the exhaustive list of options I'm now using for a full system backup (as root via sudo). Note that {{my_exclusions}} is an exercise for the reader (I'm still having some trouble getting that to work as it should). Also note the use of --link-dest... if that directory exists on the target, hard links will be made wherever to files already present in that previous backup which are unchanged in the current one.

rsync \
  --recursive \
  --links \
  --hard-links \
  --perms \
  --acls \
  --xattrs \
  --owner \
  --group \
  --devices \
  --specials \
  --times \
  --one-file-system \
  --partial \
  --numeric-ids \
  --compress \
  --delete-during \
  --stats \
  --human-readable \
  --progress \
  --verbose \
  --super \
  --log-file=/tmp/my.log \
  --link-dest=/backup/latest_backup \
  --exclude={{my_exclusions}} \
  --link-dest=/backup/latest \
  / \
  {{remote_hostname_or_ip}}:/backup/new_backup
Improve this answer
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .