11

I've got a Debian machine on my LAN serving as a backup server for the others. It has four HDDs combined into a software RAID 5 md device, on that an LVM, and on that btrfs. Backups are made using rsync, and for a large filesystem take over an hour. For a long time I had thought there would be little I could do about this.

Recently, however, I noticed that the HDD activity was very different on both ends of the transfer. While the sending side, running Gentoo and mostly using ext4, had hardly any disk IO at all, the receiving side was constantly busy. As most data wouldn't change between transfers, I believe that metadata reads should make up the bulk of the data. But I'd be really surprised if reading inodes in btrfs is so much work than doing the same in ext4.

iotop confirmed disk reads of about 1-4 MB/s on the receiving side, whereas the sending side had only the occasional 0.5 MB/s burst.

My question is, can anyone explain what's going on here? Preferably with some indication how to work around the problem if possible.

Perhaps there is some btrfs tuning flag I could use, or something similar. I need a FS with snapshot capabilities on the backup server, and my attempt to use FreeBSD and ZFS quickly lead to an inconsistent FS, so I see little alternative to btrfs at the moment. Therefore answers telling me to use ext4 or zfs might receive upvotes but no check mark.

Rsync options in use, as requested by cjm:

--rsync-path='rsync --fake-super'
--archive               # -rlptgoD
--hard-links            # detect and preserve these
--acls
--xattrs
--sparse
--noatime               # based on patch from samba #7249c1
--delete
--delete-delay
--fuzzy
--human-readable        # size suffixes, base 1000
--stats

As well as a bunch of -f rules to omit some files.

The mount options of the btrfs are reported by mount as

rw,nosuid,noexec,noatime,nospace_cache

In particular, this includes the noatime flag, so there shouldn't be any writing involved unless there actually were differences in some files. I added this information in response to the answer by Kyle Jones.

8
  • What rsync options are you using?
    – cjm
    Jul 23, 2012 at 16:53
  • Just taking a shot in the dark, do you have a failed disk? This could cause extra I/O because it's trying to reconstruct the missing data from parity information.
    – bahamat
    Jul 23, 2012 at 22:37
  • @bahamat, I've got smartd running, and it reported no issues. Neither did mdadm report any events.
    – MvG
    Jul 24, 2012 at 7:28
  • It's really tough to say for certain what's wrong. Just one example is mismatched block sizes among the layers. In order to diagnose it you're best bet is to use something like dtrace or systemtap to find out where the time is being spent.
    – bahamat
    Jul 24, 2012 at 18:42
  • @bahamat, that's an avenue I haven't investigated yet. Can you write an answer about how to use these tools to diagnose the issue? That would be great. Step-by-step instructions if you have the time, but even some rough ideas and documentation pointers would be very helpful.
    – MvG
    Jul 24, 2012 at 20:40

3 Answers 3

Reset to default
3

One possible answer is that the remote filesystem is mounted by default with the "atime" option. Access time writes for everything that the remote rsync accesses combined with the write penalty you suffer with RAID 5 (computing parity means reading all the RAID disks before you write to one of them) could explain the I/O magnification on the remote side.

If I'm right, you can speed things up by mounting the remote filesystem with the "noatime" option.

1
  • 2
    Good thought, but sadly not the solution: the filesystem already is mounted noatime. Mount reports the set of all mount options as rw,nosuid,noexec,noatime,nospace_cache.
    – MvG
    Jul 23, 2012 at 19:57
1

I suspect the --fake-super options. That tells rsync to store all the metadata information in extended attributes on each file. I suspect accessing those attributes are slow. Try a test run with rsync to a root without --fake-super. You can't reuse the same backup since the attributes won't match.

1
  • You should consider expanding upon your answer to include some helpful links or references to documentation that support your assertion.
    – HalosGhost
    Jul 1, 2014 at 17:33
1

--xattrs / -X was extremely slow before an upstream commit (still not in a release) that was picked into Debian's rsync 3.1.2-2:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=799143#20

https://bugzilla.samba.org/show_bug.cgi?id=5324

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .