I am wondering how would one design a ZFS system to build something that is massively scalable, along the lines of your favorite cloud storage, but with all the neat benefits the come with ZFS.

A more specific scenario that I am wondering about is when you've hit the maximum number of drives in your server (internal/DAS/SAN whatever) and you need to expand to another server, how, if it is indeed possible, would you create a zpool/account that would extend from one to the other, where part of the data is stored on server one and the rest on server 2? How would you even manage this?

I was particularly considering FreeBSD, because I also do not know of a way to create a server farm/cluster with FreeBSD.


ZFS is not a clustered file system, so what you talk about wouldn't actually work as described. In order to increase the capacity of a server's storage, you'd need to add more disks to the host, either in the chassis, or, if that's full, in a storage array. A host running ZFS can have up to 2^64 zpools, and each zpool can contain up to 2^64 vdevs, so you are highly unlikely, this side of doomsday, to reach that limit!

If that's not an option, and you want to integrate a new host into your infrastructure, you can export your ZFS file systems using NFS, and then mount them so that each host has the same view of the network. This way, no matter which storage box you hit, you'll get the right stuff back.

This is not clustering, and does have some overhead in terms of network traffic (although that can be somewhat mitigated by using automount to save protocol chatter from cluttering the network on quiescent file systems). However, with the guarantees that ZFS makes about data integrity, and the redundancy provided by RAIDZ, I think it's a great way to go about setting up a high capacity storage system, and is in fact how we do it at work, where we have dozens of machines exporting hundreds of TB of storage. A user can log in to any machine, and can use the same path to get to their data, giving the impression of a clustered storage setup. Exporting datasets with SAMBA means that Windows users can gain access to the data, as well.

  • event though the theoretical limit is as you so poetically stated (I am so using that expression when the chance arises :) ), in a real environment, you'd hit a physical limit; granteed NFS (perhaps CIFS :shudder:) will alleviate the issue for physical machine count, but I'm trying to see how this can be made dynamic. I was reading up on this last night and I came across an (uncomplete) response with regards to using HAST as the underlying layer with ZFS atop that. There were those who mentioned Gluster as well. It all seems very "hacky" to me. – SteveMustafa Nov 7 '13 at 21:12
  • @SteveMustafa Yes, the current solutions for FreeBSD are a bit hacky, although I like the look of ZFS on HAST - thanks for mentioning it. The HAST framework was written by PJD, so you know it'll be very well thought out and stable. I think I'm going to take a closer look at it. I don't think Gluster has been ported to FreeBSD, but it is available for NetBSD. – D_Bye Nov 8 '13 at 8:55
  • 1
    hmmm I've never actually tried HAST before, but it would seem to me that all the layering that is happening is still blech. If you do test out the HAST thing before I do, PLEASE let me know how that went. Post it as a serial on a blog or even feel free to email me. – SteveMustafa Nov 8 '13 at 16:57
  • 1
    I just realized a few things; HAST has a 2 node maximum and it acts as RAID-1, to my knowledge, there is not multi-node disk-based clustering solution that I know of on FreeBSD which is a freaking travesty IMHO. – SteveMustafa Nov 8 '13 at 23:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.