You have read the ZFS Tuning Guide, right?
Early versions of ZFS had more than a few issues with memory management, especially when using the dedupe and/or compression options. For the longest time, a simple recipe for ZFS hangs was to set compression=gzip and start copying data. Later versions of ZFS are much better. If you can, you should upgrade to FreeBSD 8.4 or 9.1.
I have a very comparable system at home (16GB RAM, 2 x 3TB SATA disks in ZFS mirror) that's been rock solid since upgrading to FreeBSD 9.1. I have compression enabled and these settings in /boot/loader.conf:
# attempts to get ZFS more stable
Another thing to check is your motherboard settings for the SATA ports. Initially, my motherboard had the SATA ports set to IDE mode and performance under ZFS was awful, making the memory exhaustion issues worse. After switching them to AHCI mode, the disks perform quite well.