7

We have this setup on our server:

  • Ubuntu 14.04 (used to be 12.04)
  • 10TB RAID-6 system
  • LVM (one VG, two LV)
  • ext4 partition ( ~2TB)
  • Btrfs partition ( ~8TB)

After a reboot last month the system was slow. First we thought it was because the RAID was resyncing (one drive was not added and was reactivated). But after that finally finished (12-13 days) access to Btrfs still is noticably slow. ext4 access seems normal.

The sysadmin who set this up (and who left this summer) already used autodefrag,noatime on the Btrfs mount.

What can we do to get Btrfs speed up again?

9

I can't comment, sorry if this is not good as answer. You should check your drives. I think 12 days of rsync time for 10Tb is too long, should be more like 12-24 hours. Look at the different drives with smartctl to check if one has many errors:

for i in a b c d e f g h i j k l; do echo $i ; smartctl -x /dev/sd$i | grep occurred | head -1 ; done

I have seen working RAID slow down because of this. And IIRC Btrfs needs more disc access than ext4 for directory look ups, which could explain access speed difference.

  • 1
    Thanks so much for this tip. I used it and found one drive with 832 errors (and one with 2, no errors on the rest). After power down and removing that drive Btrfs access has sped up noticeable. A replacement drive is on its way. – RachelP83 Dec 31 '14 at 10:02
  • I just logged back into the server from home: the resync of the new drive has already finished! Directory traversal speeds seem back to normal. – RachelP83 Jan 1 '15 at 15:54
4

You've just stacked three layers of disk I/O indirection up, and you're wondering why it performs badly?

There's an old saying in computer science, attributed to David Wheeler:

Any problem in computer science can be solved with another layer of indirection.

It's not true. There is one problem you cannot solve that way: the system is too slow. (Not without adding more hardware anyway. More spindles in this case, or clustering, or load balancing, etc.)

Btrfs includes LVM functionality, but you've gone and stacked that on top of Linux's LVM2 layer and put that on top of a hardware RAID-6 system. What else have you done, hosted virtual machines on this teetering pile of abstraction, so that you can add another layer or two of filesystem indirection?

You've asked for advice, so here's mine: snap some of those layers.

If it were me, I'd configure the hardware disk subsystem for JBOD and lay Btrfs directly over it.

More broadly, you should not expect Btrfs to achieve the same performance as ext[234]. You aren't comparing apples to apples. Btrfs buys you a higher level of data integrity, so it will naturally run a bit slower. TANSTAAFL.

  • Your observations are appreciated, but I cannot change the hardware right now, and having only Btrfs caused problems with certain large files that were updated (a known problem), so we cannot have that directly "on the metal". Anyway performance used to be good, with layers and in place. – RachelP83 Dec 31 '14 at 7:55
  • 5
    In my experience, btrfs does a lot of housekeeping when a filesystem is mounted, also depending on whether it was cleanly unmounted or not. Are any kernel processes related to btrfs running in state 'D' when you run ps aux? If so just wait it out. Don't reboot, it'll restart the action... Apart from that, LVM hardly incurs any performance penalty; at least I couldn't measure it in my unscientific way. – wurtel Dec 31 '14 at 7:59
  • 3
    Note that information about LVM volume is loaded into device mapper which is part of block IO layer in the kernel. This means that the performance impact of lvm alone should be hardly noticeable. – marbu Feb 18 '15 at 9:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.