What are the options to create a single volume of multiple different hard drives with redundancy (i.e. heterogeneous RAID)

Question

I have about 8 hard drives lying around, and enough SATA ports in my main PC (running openSUSE Tumbleweed) to connect them all. Some of the drives are more than 5 years old, and the sizes range from 384GB to 2TB.

I think it'd be a shame to let all that storage space go unused, so I'd like to install and use them, but obviously not as separate volumes. I used to run mdadm RAID5 and RAID6 arrays with older disks, but in this case, that would become tedious (one 8-drive array of 384GB per disk, one 7-drive array with the difference to the next-larger ... and so on). So I've been looking for alternatives.

I've also been running a BTRFS RAID5 array on 4 identical drives for a while now, and given my experience with that (about one full restore per year needed, with some files lost...), and the fact that even the developers don't seem to expect they're going to make it work reliably without users having to stay very alert makes me doubt that BTRFS for a heterogeneous array like this would be a wise choice.

I've heard miracoulous things about ZFS and its RAIDZ feature being able to accommodate different-sized drives with even 3 parity blocks but most of the advice I've been able to find so far only deals with identical drives, or at least identically-sized RAID partitions.

My priority is neither speed nor access times as the machine is only used by me and hard drive throughput is very rarely a concern. My intention is to use the space on those hard drives in relative comfort for the remainder of their lives, and not have to worry about the occasional drive failure costing me data or copious amounts of time. I'd like to have some setup where I can just bundle them all together and if one of the drives fails, I'd remove it and carry on (or maybe add a new one if I need the space).

I don't mind doing some extra up-front configuration as long as I can rest assured that it does not need my personal attention unless a drive breaks, and that it will let me know if/when that happens. Having 2 or 3 drive failures covered would be nice but I'd also be fine if I could use half the capacity (i.e. RAID10-like), as long as I can make use of all the drives, and most of the space on them.

1. Does something like this exist?
2. What are the options?
3. Is there a how-to that explains how to set things up?

Answer 1

The key to accomplishing what you are seeking is to group your drives into sets that total to the same or nearly the same amount of storage, and then to concatenate each set of physical drives into a single, virtual drive that represents the sum total of the storage available from the drives in that set.

I'll do this in FreeBSD, but I presume that similar tools exist in Linux as well. The primary tool at hand here is gconcat(8).

Let's consider a scenario where you have:

• five 384G drives (as /dev/md0-md4)
• four 2T drives (as /dev/md5-md8)
• two 1T drives (as /dev/md9-md10)

Using gconcat we can contenate these drives into five sets:

• four sets of 2T + 384G
• one set of 1T + 1T + 384G

We'll then have 5 similarly-sized virtual concatenated disks, and can build our pool from those. To save space, I'll actually de-rate those sizes by a factor of 2^10, but the concept remains the same. So I'll use 5 drives of 384M, 4 drives of 2G, and 2 drives of 1G.

First, given the /dev/md* devices outlined above (which may be /dev/sd* or similar for physical disks in Linux), we can concatenate those physical disks into five virtual disks that we will name vdisk0-vdisk4:

# gconcat create -v vdisk0 md0 md5
# gconcat create -v vdisk1 md1 md6
# gconcat create -v vdisk2 md2 md7
# gconcat create -v vdisk3 md3 md8
# gconcat create -v vdisk4 md4 md9 md10

In FreeBSD, these concatenated disks show up as devices /dev/concat/vdisk0-vdisk4. As with many FreeBSD commands built around disk device manipulation, the /dev/ is understood and is omitted for brevity in the output of gconcat status. Likewise, GEOM-based commands that reference /dev/ entries can omit the leading /dev/ for convenience.

# gconcat status
         Name  Status  Components
concat/vdisk0      UP  md0
                       md5
concat/vdisk1      UP  md1
                       md6
concat/vdisk2      UP  md2
                       md7
concat/vdisk3      UP  md3
                       md8
concat/vdisk4      UP  md4
                       md9
                       md10

We can also verify that the mediasize of each concatenated disk is similar (or in this example, identical):

# gconcat list | grep -A2 ^Prov
Providers:
1. Name: concat/vdisk0
   Mediasize: 2550136832 (2.4G)
--
Providers:
1. Name: concat/vdisk1
   Mediasize: 2550136832 (2.4G)
--
Providers:
1. Name: concat/vdisk2
   Mediasize: 2550136832 (2.4G)
--
Providers:
1. Name: concat/vdisk3
   Mediasize: 2550136832 (2.4G)
--
Providers:
1. Name: concat/vdisk4
   Mediasize: 2550136832 (2.4G)

So! We now have five devices of 2.4G each and can create a ZFS pool from them. Let's create a double-parity RAIDZ2 pool, but set its mountpoint to none:

# zpool create -m none tank raidz2 concat/vdisk{0..4}
# zpool status tank
  pool: tank
 state: ONLINE
config:

    NAME               STATE     READ WRITE CKSUM
    tank               ONLINE       0     0     0
      raidz2-0         ONLINE       0     0     0
        concat/vdisk0  ONLINE       0     0     0
        concat/vdisk1  ONLINE       0     0     0
        concat/vdisk2  ONLINE       0     0     0
        concat/vdisk3  ONLINE       0     0     0
        concat/vdisk4  ONLINE       0     0     0

errors: No known data errors
# zfs list tank
NAME   USED  AVAIL     REFER  MOUNTPOINT
tank   179K  6.64G     39.3K  none

QED.

In summary, the key concept is to group your drives into sets that total as nearly as possible to the same amount of storage, because the smallest set of drives will limit your pool's VDEV size. Once you have an optimal grouping of your physical drives, concatenate each set of drives into a virtual disk that represents the combined storage of that set. Finally, build your ZPOOL from the virtual concatenated drives.

What if a drive fails?

In case of a drive failure -- let's use /dev/concat/vdisk2 as an example -- simply destroy the concatenated virtual disk whose physical drive(s) have failed, replace the failed drive(s), recreate the virtual disk, and then resilver the pool onto the concatenated disk.

# zpool status tank
  pool: tank
 state: DEGRADED
status: One or more devices has been removed by the administrator.
    Sufficient replicas exist for the pool to continue functioning in a
    degraded state.
action: Online the device using zpool online' or replace the device with
    'zpool replace'.
config:

    NAME               STATE     READ WRITE CKSUM
    tank               DEGRADED     0     0     0
      raidz2-0         DEGRADED     0     0     0
        concat/vdisk0  ONLINE       0     0     0
        concat/vdisk1  ONLINE       0     0     0
        concat/vdisk2  REMOVED      0     0     0
        concat/vdisk3  ONLINE       0     0     0
        concat/vdisk4  ONLINE       0     0     0

errors: No known data errors

Since vdisk2 has failed, we destroy it:

# gconcat destroy vdisk2

Next, we replace the failed drive(s) that comprise concat/vdisk2. If you have to use alternate drive sizes, ensure that the new vdisk2 is not smaller than the smallest vdisk in the pool, or ZFS will not let you use that disk in the pool.

Once the failed drives are exchanged for suitable replacements, re-concatenate those physical drives to create the new vdisk2:

# gconcat create vdisk2 md2 md7
# zpool online tank concat/vdisk2
# zpool status tank
  pool: tank
 state: ONLINE
  scan: resilvered 7.50K in 00:00:01 with 0 errors on Wed Sep 22 11:26:55 2021
config:

    NAME               STATE     READ WRITE CKSUM
    tank               ONLINE       0     0     0
      raidz2-0         ONLINE       0     0     0
        concat/vdisk0  ONLINE       0     0     0
        concat/vdisk1  ONLINE       0     0     0
        concat/vdisk2  ONLINE       0     0     0
        concat/vdisk3  ONLINE       0     0     0
        concat/vdisk4  ONLINE       0     0     0

errors: No known data errors