FreeBSD 10 with Root-on-ZFS using only a part of the disk

Question

I'm having a bit of trouble here. I'll try and be as concise as possible in describing my objective:

I'm setting up a storage server for my employer, using two Intel DC S3500 series 120GB solid state drives (and 12 x 2TB SAS drives for storage). I would like to install freebsd10 with root-on-zfs on these drives, but only use a small (12GB) partition to create a zfs mirror for the OS. The remaining space, I want to use for a ZIL/SLOG (24GB), and the rest for a L2ARC (striped, everything that's left, on both drives).

I cannot figure out how to tell the freebsd10 installer to only use 'ada0p2' and 'ada1p2' as my mirror for installing the OS onto. I only get the option for using the entire drives.

Additionally, I've tried installing as normal and then using zfs send and recv to backup, resize the partition with gpart, and then restore a replica stream of my zroot pool... the system refuses to boot when I do that.

How do I go about achieving this goal? Any assistance with this would be much appreciated.

Answer 1

I don't think the installer can do what you want yet (although it's getting better over time), so you could try booting the installation image, and run a root shell from the initial menu. You can then use gpart, zpool and zfs to configure your disks by hand and install the system from the archives on the image.

There are numerous guides around the Internet, but I find that Matthew Seaman's is the best for my needs. It describes a mirrored root-on-zfs setup that supports boot environments (I use a slightly modified version of the sysutils/beadm port to manage my boot environments). It doesn't talk about configuring log and cache devices, but it should give you enough information to get the OS installed as you want it, and you can then add logging and cache devices after the fact.

There are also some good resources linked from the RootOnZFS page on the FreeBSD wiki.

Whichever guide you decide to follow, personal experience suggests that you allow yourself time to run through it a couple of times to get the feel for it and to ensure you understand your config, before you commit the box to a production environment.

Answer 2

I haven't used the new installer yet but I have used mfsbsd with 9.x, doing exactly what you describe.

There is an option to the zfsinstall on mfsbsd:

-z zfs_part_size  : create zfs parition of this size (default: all space left)

mfsbsd is really simple and fast to use.

Answer 3

Here's the solution I used to the problem, for future reference:

I installed freebsd 10.0-RELEASE using the "stock" disc1 ISO, selected root-on-zfs and targeted the two SSDs I wanted to use. I completed the installation, then booted into the live-cd environment, inserted a small USB flash drive made a filesystem on it, made a zroot@fresh snapshot, and used zfs send -R zroot@fresh > /path/to/removable/zroot.bin to create a replica stream backup of the snapshot. For good measure, I also saved a copy of all the zroot pool attributes with zpool get all zroot > /path/to/removable/pool.txt, since this is not included in the zfs send / recv process.

The next step is to export (or destroy) zroot so that it's no longer alive which lets us use gpart to shrink the partitions down. Then create a new pool on the shrunk down partitions, specifying the altroot as somewhere writable in the livecd environment such as /mnt or /tmp. Next, recv the replica stream back to your recreated zroot pool: zfs recv -Fdv zroot < /path/to/removable/zroot.bin

The key area where I was getting thrown off is that all the guides online that instruct you to tell zfs how to boot from the pool use zpool set bootfs=zroot/boot or something similar. However this is the wrong dataset to boot from (it doesn't exist). the bsdinstall automatic root-on-zfs process creates zroot/ROOT/default as the bootfs. Check your saved copy of the zroot pool attributes if you need to replicate any other settings. Mark the correct bootfs using: zpool set bootfs=zroot/ROOT/default zroot and everything should be gravy after that.