FreeBSD zfs timeout errors

Question

I have FreeBSD 8.2 with ZFS v28. 8 cpu cores, 16Gb RAM. Two disk drives (1.5Tb WD Green) on ZFS mirror raid.

My system strangely uses memory. With a lot of free (wired) memory, it starts to use swap and then doesn't clean it! Swap usage keeps growing. munin-graph swap

And then I see on my kvm these messages:

All file reads or writes seem very slow. When I use ssh and, for example, do nano /etc/rc.conf it pauses for 3-5 seconds before it works.

Will my disk drives die soon, or is the problem elsewhere?

Answer 1

You have read the ZFS Tuning Guide, right?

Early versions of ZFS had more than a few issues with memory management, especially when using the dedupe and/or compression options. For the longest time, a simple recipe for ZFS hangs was to set compression=gzip and start copying data. Later versions of ZFS are much better. If you can, you should upgrade to FreeBSD 8.4 or 9.1.

I have a very comparable system at home (16GB RAM, 2 x 3TB SATA disks in ZFS mirror) that's been rock solid since upgrading to FreeBSD 9.1. I have compression enabled and these settings in /boot/loader.conf:

# attempts to get ZFS more stable
vfs.zfs.arc_max="4096M"
kern.maxvnodes=250000
vfs.zfs.prefetch_disable=1
vm.kmem_size="1024M"
vm.kmem_size_max="1024M"

Another thing to check is your motherboard settings for the SATA ports. Initially, my motherboard had the SATA ports set to IDE mode and performance under ZFS was awful, making the memory exhaustion issues worse. After switching them to AHCI mode, the disks perform quite well.

Answer 2

This looks like a hardware problem - either drives or controller. Since there are errors on both ad4 and ad6, it may be the controller. I recommend backing up any important data as soon as possible.

You might try installing sysutils/smartmontools to see if any SMART errors are reported.

As to your RAM usage, that seems strange. Even if you're using ZFS v28's dedupe, I wouldn't expect a 1.5TB filesystem to take up that much RAM. You might try running top -S -osize to see what is using the memory.

Answer 3

You can check using:

• smartctl -s on -A /dev/adX

Also, check to see the raw value of Current PEnding Sector (bad sectors) and UDMA CRC Error Count (cable errors).

It might also be a problem in the legacy ATA driver ('ad') whereas the newer AHCI driver is called 'ada' and should be much better with modern code and shared CAM design. Your BIOS' controller must be set to AHCI for this to work.