ZFS on Linux and Amazon EC2

Question

I have a situation where I need dynamically expandable storage space on a Linux machine running in Amazon EC2, and ZFS came to mind. If I remember properly, ZFS supports a way to dynamically expand a non-redundant JBOD-like array.

I have a few requirements for a project I'm working on:

1. I should be able to move the array to another EC2 instance if need be.
2. I should be able to dynamically expand the ZFS array, adding new "drives" to the ZFS volume without any downtime.

Is this possible? If ZFS provides the ability to dynamically expand the array, I should be able to run a script which would dynamically create new EBS volumes, attach them to the EC2 instance, and have ZFS add them to its pool, all dynamically without downtime.

Answer 1

ZFS on Linux does not have a critical mass following, whatever that may be. ZFS will effectively lock you in. The underlying format is incompatible with rescue disks, and distributions for which you will find ZFS is rare.

I would tend to overlook these limits due to my bias towards ZFS. You may want to google for a service provider that provides SmartOS/ZFS based Cloud housting.

Linux file systems ext3/ext4 and probably various others do allow for dynamic growing. The volume manager built into Linux, LVM, also allows you to dynamically expand volumes onto new disks. For anything other than experimentation this would have to be the current best Linux recommendation, though that would change once ZFS got a critical mass following on Linux.

The process for Linux + LVM + ext3 is:

Example: LVM volume group myvg, mounted volume name uservol1, and disk device in Linux is /dev/sdf

1. Allocate the disk to the VM. (in Amazon Management Console, Create the EBS volume, write down its ID, and allocate it to the instance)
2. The EC2 instance should have some udev rules for creating the device node. So you should see a new disk in /dev/sd* ... Log in on the instance and check that the EBS volume is visible, eg fdisk -l /dev/sdf, cat /proc/partitions, run blkid.
3. Create partition table if needed: fdisk / sfdisk
4. Initialize for LVM use: pvcreate /dev/sdf
5. Add the disk (physical volume) to the LVM volume group vgextend myvg /dev/sdf
6. Grow the Volume size: lvextend -L +1024G /dev/myvg/uservol1
7. Grow the ext3/ext4 file system: resize2fs /dev/myvg/uservol1
8. Check (df -h) and you should see that the mounted file system now have more space.

QED.

Answer 2

Yes, you can most definitely auto expand the array with ZFS. You just need to set the following property:

# zpool set autoexpand=on pool
# zpool set expandsize=on pool

These properties will give you two benefits.

1. As you add any VDEV to the pool, it will be automatically expanded
2. As you replace 1TB drives with 2TB drives, when all drives are replaced, the pool will automatically resize to accommodate the new drive sizes.

There are a couple of caveats:

1. You cannot remove a drive nor can you remove a VDEV from the pool. So, while you can grow your pool, you cannot shrink it. However, you can detach a drive from a mirror only.
2. All drives should be the same specifications: same size, same speed, etc. Same make and model is not necessary.
3. All VDEVs should be the same VDEV. If you have one RAIDZ in your pool, then all VDEVs should be RAIDZ.
4. All VDEVs should be the same size. If one VDEV is 2TB. All VDEVs should be 2TB.