ZFS - two pools, one root.

Question

I have a desktop machine with two sets of disks - small SSDs for things I need to be quick and large HDDs for things I need to be big.

I have set up the small SSDs into a pool called "rpool" and successfully followed the instructions to use it as Ubuntu's root filesystem - it even boots from it. https://github.com/zfsonlinux/zfs/wiki/Ubuntu-16.04-Root-on-ZFS

ZFS will not let me create a second pool ("hpool" for the HDDs) with a root of /. To work around this I've used a root of /hdd for hpool.

This means that all of my hdd datasets are mounting under /hdd, not under / where I want them to be - e.g. /hdd/tmp instead of /tmp.

How can I fix this? (Other than by a mess of symlinks.)

Answer 1

While creating a new pool you can use the -R option to specify the alternate root. The ZFS option -O canmount=off is added to prevent that this pool might be mounted first and to prevent the following error message when creating child filesystems in the new pool:

zfs create tpool/tst
cannot mount 'tpool' on '/': directory is not empty
cannot mount 'tpool/tst' on '/tst': failure mounting parent dataset

Create the second pool as follows:

zpool create -N -R / -O canmount=off tpool device

When you create a filesystem in the new pool using:

zfs create tpool/tst

It will show as:

NAME                                USED  AVAIL  REFER  MOUNTPOINT
rpool                              15.7G  15.3G    34K  /rpool
.
tpool                               146K   984M    31K  /
tpool/tst                            31K   984M    31K  /tst

ls -l /tst/
total 0

Answer 2

Answering to the question "How do I fix this?":

All datasets have a mountpoint property. You can set it to whatever directory you like. In your case, /hdd is just the default if you don't set it. e.g.:

zfs set mountpoint=/foo hdd/foo

Some other replies so far make good points but answer to the unasked question "What should I have done to avoid this in the first place?"

Answer 3

/hdd was chosen in your configuration because this is the base mountpoint where your pool has been mounted and ZFS uses inheritance for propagating certain properties.

You can mount every ZFS dataset where you want by setting its mountpoint property.

zfs set mointpoint=/tmp pool2/tmp

Please note that the child datasets of pool2/tmp will inherit the parent mointpoint, unless you correct it for each single one.

Also be careful when remounting system directories. It is hard to recover from easily made mistakes here.

Answer 4

You're doing it wrong.

ZFS does support mixed spinning and silicon storage in a single pool, but in a different way than you're attempting. Instead of manually managing which data goes where, ZFS wants to handle this for you via its ZIL and L2ARC caches.

Set up this way, ZFS will figure out what needs to be where based on your usage patterns.

If you must have manual management over what goes where, I'd suggest simply making three pools: one each for the SSDs, and a mirror or RAID-Z for the HDDs. Use one SSD pool for the root filesystem, one for /home, and the slow HDD pool for bulk data. Trying to squish it all into a single thing yet still maintaining manual management is mixing concepts. Either you want ZFS to manage which disks your data lives on or you do not.