Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I've got a Debian machine on my LAN serving as a backup server for the others. It has four HDDs combined into a software RAID 5 md device, on that an LVM, and on that btrfs. Backups are made using rsync, and for a large filesystem take over an hour. For a long time I had thought there would be little I could do about this.

Recently, however, I noticed that the HDD activity was very different on both ends of the transfer. While the sending side, running Gentoo and mostly using ext4, had hardly any disk IO at all, the receiving side was constantly busy. As most data wouldn't change between transfers, I believe that metadata reads should make up the bulk of the data. But I'd be really surprised if reading inodes in btrfs is so much work than doing the same in ext4.

iotop confirmed disk reads of about 1-4 MB/s on the receiving side, whereas the sending side had only the occasional 0.5 MB/s burst.

My question is, can anyone explain what's going on here? Preferably with some indication how to work around the problem if possible.

Perhaps there is some btrfs tuning flag I could use, or something similar. I need a FS with snapshot capabilities on the backup server, and my attempt to use FreeBSD and ZFS quickly lead to an inconsistent FS, so I see little alternative to btrfs at the moment. Therefore answers telling me to use ext4 or zfs might receive upvotes but no check mark.


Rsync options in use, as requested by cjm:

    --rsync-path='rsync --fake-super'
    --archive               # -rlptgoD
    --hard-links            # detect and preserve these
    --acls
    --xattrs
    --sparse
    --noatime               # based on patch from samba #7249c1
    --delete
    --delete-delay
    --fuzzy
    --human-readable        # size suffixes, base 1000
    --stats

As well as a bunch of -f rules to omit some files.


The mount options of the btrfs are reported by mount as

rw,nosuid,noexec,noatime,nospace_cache

In particular, this includes the noatime flag, so there shouldn't be any writing involved unless there actually were differences in some files. I added this information in response to the answer by Kyle Jones.

share|improve this question
    
What rsync options are you using? –  cjm Jul 23 '12 at 16:53
    
Just taking a shot in the dark, do you have a failed disk? This could cause extra I/O because it's trying to reconstruct the missing data from parity information. –  bahamat Jul 23 '12 at 22:37
    
@bahamat, I've got smartd running, and it reported no issues. Neither did mdadm report any events. –  MvG Jul 24 '12 at 7:28
    
It's really tough to say for certain what's wrong. Just one example is mismatched block sizes among the layers. In order to diagnose it you're best bet is to use something like dtrace or systemtap to find out where the time is being spent. –  bahamat Jul 24 '12 at 18:42
    
@bahamat, that's an avenue I haven't investigated yet. Can you write an answer about how to use these tools to diagnose the issue? That would be great. Step-by-step instructions if you have the time, but even some rough ideas and documentation pointers would be very helpful. –  MvG Jul 24 '12 at 20:40

2 Answers 2

One possible answer is that the remote filesystem is mounted by default with the "atime" option. Access time writes for everything that the remote rsync accesses combined with the write penalty you suffer with RAID 5 (computing parity means reading all the RAID disks before you write to one of them) could explain the I/O magnification on the remote side.

If I'm right, you can speed things up by mounting the remote filesystem with the "noatime" option.

share|improve this answer
2  
Good thought, but sadly not the solution: the filesystem already is mounted noatime. Mount reports the set of all mount options as rw,nosuid,noexec,noatime,nospace_cache. –  MvG Jul 23 '12 at 19:57

I suspect the --fake-super options. That tells rsync to store all the metadata information in extended attributes on each file. I suspect accessing those attributes are slow. Try a test run with rsync to a root without --fake-super. You can't reuse the same backup since the attributes won't match.

share|improve this answer
    
You should consider expanding upon your answer to include some helpful links or references to documentation that support your assertion. –  HalosGhost Jul 1 at 17:33

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.