I am planning a storage server where users will store up to 20 TB of data. Since I have made some good experiences with ZFS on Linux, I would like to use that. However, I know that the amount of data will grow by several 100 GB per year, so at some point, I will need to increase my pool size. Since there is so much data, I am afraid that it will be quite a hassle to completely destroy the zpool and recreate it; so I wonder whether it would be possible to increase the pool capacity by adding more disks and keeping the existing ones. The pool will be RAIDZ-1 at least. Or is it true that it will be necessary to remove one disk after another and replace it with a larger one? no, I cannot believe that. How do larger servers than mine handle the increasing demand for more storage capacity?
There are basically two ways of growing a ZFS pool.
Add more vdevs
This is what user1133275 is suggesting in their answer. It's done with
zpool add (which has basically the same syntax as
zpool create does for specifying storage), and it works well for what it does.
ZFS won't rebalance your stored data automatically, but it will start to write any new data to the new vdevs until the new vdev has about the same usage as the existing one(s).
Once you've added a vdev to a pool, you basically cannot remove it without recreating the pool from scratch.
All vdevs in a pool need to be above their respective redundancy thresholds for the pool to be importable. In other words, every vdev needs to be at least
DEGRADED for the pool to function.
Replace disks with larger ones
This is what you're discussing in your question. It's the normal way of growing a ZFS pool when you have a pool layout that you are happy with.
To replace a device with a new one, the new device needs to be at least as large as the old one.
Operationally, you'd hook up the new disk along with the old, and then
zpool replace the old disk with the new one. (This creates a temporary
replacing device which becomes a parent to the old and new disk; when the resilver completes, the
replacing device is removed from the device tree and it looks like the new device was there all along.) Once the resilver completes, the old disk can be removed from the system.
Once all disks in a vdev are replaced by larger ones, you can expand the pool by running
zpool online -e or by having the
autoexpand property set to
on (though I wouldn't really recommend the latter; pool expansion should be a conscious decision).
So which way is better?
That basically depends on your pool. As mentioned, the downside to having multiple vdevs is that they all need to be functional, so by adding vdevs you are actually, in a sense, reducing your safety margin. The upside, though, is that it's much easier to grow the pool piecemeal. Replacing devices in-place is basically the opposite; you don't need to keep as many vdevs functioning, but it isn't as easy to grow a pool piecemeal.
For me, frankly, assuming for a second that you're using rotational hard disks (since this seems like bulk storage), 20 TB is still well within reason for a single vdev pool. My suggestion in your situation would be to get six drives of the 8 TB variety, and to set those up in a single raidz2 vdev. Doing so gives you a net storage capacity of around 32 TB, thus leaving you with an initial about 35% free, and the ability to lose any two drives before any of your data is at significant risk. You could also consider running eight 6 TB drives for a net storage capacity of around 36 TB and starting out at 45% free. (I'd consider 6-8 drives to be slightly on the large end for raidz1, but fine for raidz2.) Then plan to replace those drives either on a 4-5 year schedule (due to wear) or whenever the pool goes above about 80% full (because ZFS is much, much happier when it has good headroom). If your figures are accurate, you should be replacing those drives due to wear well before your pool starts getting full, while still allowing for a reasonable amount of unexpected growth in storage needs. When you replace the drives, you can decide whether you're happy with the pool size you've got based on then-current usage, or if you want to get larger drives and expand the pool.
In addition to the options in the excellent answer above, there will soon be another option.
The OpenZFS project (ZFS on Linux, ZFS on FreeBSD) is working on a feature to allow the addition of new physical devices to existing RAID-Z vdevs. This will allow, for instance, the expansion of a 6-drive RAID-Z2 vdev into a 7-drive RAID-Z2 vdev. This will happen while the filesystem is online, and will be repeatable once the expansion is complete (e.g., 7-drive vdev → 8-drive vdev).
As of September 2020, this feature is still in development: https://github.com/openzfs/zfs/pull/8853
Adding space is easy in zfs ( and btrfs, and lvm2)
zpool add $POOL mirror $A $B