1

I have about 8 hard drives lying around, and enough SATA ports in my main PC (running openSUSE Tumbleweed) to connect them all. Some of the drives are more than 5 years old, and the sizes range from 384GB to 2TB.

I think it'd be a shame to let all that storage space go unused, so I'd like to install and use them, but obviously not as separate volumes. I used to run mdadm RAID5 and RAID6 arrays with older disks, but in this case, that would become tedious (one 8-drive array of 384GB per disk, one 7-drive array with the difference to the next-larger ... and so on). So I've been looking for alternatives.

I've also been running a BTRFS RAID5 array on 4 identical drives for a while now, and given my experience with that (about one full restore per year needed, with some files lost...), and the fact that even the developers don't seem to expect they're going to make it work reliably without users having to stay very alert makes me doubt that BTRFS for a heterogeneous array like this would be a wise choice.

I've heard miracoulous things about ZFS and its RAIDZ feature being able to accommodate different-sized drives with even 3 parity blocks but most of the advice I've been able to find so far only deals with identical drives, or at least identically-sized RAID partitions.

My priority is neither speed nor access times as the machine is only used by me and hard drive throughput is very rarely a concern. My intention is to use the space on those hard drives in relative comfort for the remainder of their lives, and not have to worry about the occasional drive failure costing me data or copious amounts of time. I'd like to have some setup where I can just bundle them all together and if one of the drives fails, I'd remove it and carry on (or maybe add a new one if I need the space).

I don't mind doing some extra up-front configuration as long as I can rest assured that it does not need my personal attention unless a drive breaks, and that it will let me know if/when that happens. Having 2 or 3 drive failures covered would be nice but I'd also be fine if I could use half the capacity (i.e. RAID10-like), as long as I can make use of all the drives, and most of the space on them.

  1. Does something like this exist?
  2. What are the options?
  3. Is there a how-to that explains how to set things up?
8
  • 1
    I'm about to migrate my NAS off zfs and back to raid+ext4 as I can't afford the unpredictable kernel memory overheads of zfs and the subsequent lockups
    – roaima
    May 3, 2021 at 15:50
  • 1
    It's an HP ProLiant Microserver (Gen8) with 16GB memory and four disk slots. I am running a couple of VMs on the system too, but I don't like the way ZFS seems to "borrow" a couple of GB every so often without being able to swap out user mode processes (if the RAM isn't immediately available you get a lockup)
    – roaima
    May 3, 2021 at 20:08
  • 1
    @roaima try setting zfs_arc_min and zfs_arc_max to the same value. e.g. for 4GB ARC, options zfs zfs_arc_min=4294967296 zfs_arc_max=4294967296 in /etc/modprobe.d/zfs and run update-initramfs. Or use whatever method you currently use to pass options to kernel modules on boot.
    – cas
    May 4, 2021 at 12:07
  • 1
    @zak ZFS can use drives of different sizes, but the resulting vdev will be based on the size of the smallest drive. e.g. if you create a mirror vdev with a 1TB drive and a 2TB drive, the vdev will be 1TB (btw, if you later replace the 1TB with a 2TB, the vdev will automatically expand to 2TB). A pool can contain multiple vdevs so try to add vdevs built from same or similar sized drives. btrfs doesn't have all the features or reliability of ZFS but it is capable of using mismatched drives at their full capacity.
    – cas
    May 4, 2021 at 12:13
  • 1
    @roaima i've never seen a zfs box crash as you describe from low memory. I have seen the kernel OOM killer kill random processes when the machine runs out of memory (due to, e.g., chromium or firefox. or VMs). I've seen that on machines with and without ZFS pools, seems to work exactly the same in my experience. If you're running out of memory, I suggest adding more - even if it's ancient DDR3 stuff, it's easy to find a cheap 8 or 16 GB.
    – cas
    May 4, 2021 at 12:18

1 Answer 1

1

The key to accomplishing what you are seeking is to group your drives into sets that total to the same or nearly the same amount of storage, and then to concatenate each set of physical drives into a single, virtual drive that represents the sum total of the storage available from the drives in that set.

I'll do this in FreeBSD, but I presume that similar tools exist in Linux as well. The primary tool at hand here is gconcat(8).

Let's consider a scenario where you have:

  • five 384G drives (as /dev/md0-md4)
  • four 2T drives (as /dev/md5-md8)
  • two 1T drives (as /dev/md9-md10)

Using gconcat we can contenate these drives into five sets:

  • four sets of 2T + 384G
  • one set of 1T + 1T + 384G

We'll then have 5 similarly-sized virtual concatenated disks, and can build our pool from those. To save space, I'll actually de-rate those sizes by a factor of 2^10, but the concept remains the same. So I'll use 5 drives of 384M, 4 drives of 2G, and 2 drives of 1G.

First, given the /dev/md* devices outlined above (which may be /dev/sd* or similar for physical disks in Linux), we can concatenate those physical disks into five virtual disks that we will name vdisk0-vdisk4:

# gconcat create -v vdisk0 md0 md5
# gconcat create -v vdisk1 md1 md6
# gconcat create -v vdisk2 md2 md7
# gconcat create -v vdisk3 md3 md8
# gconcat create -v vdisk4 md4 md9 md10

In FreeBSD, these concatenated disks show up as devices /dev/concat/vdisk0-vdisk4. As with many FreeBSD commands built around disk device manipulation, the /dev/ is understood and is omitted for brevity in the output of gconcat status. Likewise, GEOM-based commands that reference /dev/ entries can omit the leading /dev/ for convenience.

# gconcat status
         Name  Status  Components
concat/vdisk0      UP  md0
                       md5
concat/vdisk1      UP  md1
                       md6
concat/vdisk2      UP  md2
                       md7
concat/vdisk3      UP  md3
                       md8
concat/vdisk4      UP  md4
                       md9
                       md10

We can also verify that the mediasize of each concatenated disk is similar (or in this example, identical):

# gconcat list | grep -A2 ^Prov
Providers:
1. Name: concat/vdisk0
   Mediasize: 2550136832 (2.4G)
--
Providers:
1. Name: concat/vdisk1
   Mediasize: 2550136832 (2.4G)
--
Providers:
1. Name: concat/vdisk2
   Mediasize: 2550136832 (2.4G)
--
Providers:
1. Name: concat/vdisk3
   Mediasize: 2550136832 (2.4G)
--
Providers:
1. Name: concat/vdisk4
   Mediasize: 2550136832 (2.4G)

So! We now have five devices of 2.4G each and can create a ZFS pool from them. Let's create a double-parity RAIDZ2 pool, but set its mountpoint to none:

# zpool create -m none tank raidz2 concat/vdisk{0..4}
# zpool status tank
  pool: tank
 state: ONLINE
config:

    NAME               STATE     READ WRITE CKSUM
    tank               ONLINE       0     0     0
      raidz2-0         ONLINE       0     0     0
        concat/vdisk0  ONLINE       0     0     0
        concat/vdisk1  ONLINE       0     0     0
        concat/vdisk2  ONLINE       0     0     0
        concat/vdisk3  ONLINE       0     0     0
        concat/vdisk4  ONLINE       0     0     0

errors: No known data errors
# zfs list tank
NAME   USED  AVAIL     REFER  MOUNTPOINT
tank   179K  6.64G     39.3K  none

QED.

In summary, the key concept is to group your drives into sets that total as nearly as possible to the same amount of storage, because the smallest set of drives will limit your pool's VDEV size. Once you have an optimal grouping of your physical drives, concatenate each set of drives into a virtual disk that represents the combined storage of that set. Finally, build your ZPOOL from the virtual concatenated drives.

What if a drive fails?

In case of a drive failure -- let's use /dev/concat/vdisk2 as an example -- simply destroy the concatenated virtual disk whose physical drive(s) have failed, replace the failed drive(s), recreate the virtual disk, and then resilver the pool onto the concatenated disk.

# zpool status tank
  pool: tank
 state: DEGRADED
status: One or more devices has been removed by the administrator.
    Sufficient replicas exist for the pool to continue functioning in a
    degraded state.
action: Online the device using zpool online' or replace the device with
    'zpool replace'.
config:

    NAME               STATE     READ WRITE CKSUM
    tank               DEGRADED     0     0     0
      raidz2-0         DEGRADED     0     0     0
        concat/vdisk0  ONLINE       0     0     0
        concat/vdisk1  ONLINE       0     0     0
        concat/vdisk2  REMOVED      0     0     0
        concat/vdisk3  ONLINE       0     0     0
        concat/vdisk4  ONLINE       0     0     0

errors: No known data errors

Since vdisk2 has failed, we destroy it:

# gconcat destroy vdisk2

Next, we replace the failed drive(s) that comprise concat/vdisk2. If you have to use alternate drive sizes, ensure that the new vdisk2 is not smaller than the smallest vdisk in the pool, or ZFS will not let you use that disk in the pool.

Once the failed drives are exchanged for suitable replacements, re-concatenate those physical drives to create the new vdisk2:

# gconcat create vdisk2 md2 md7
# zpool online tank concat/vdisk2
# zpool status tank
  pool: tank
 state: ONLINE
  scan: resilvered 7.50K in 00:00:01 with 0 errors on Wed Sep 22 11:26:55 2021
config:

    NAME               STATE     READ WRITE CKSUM
    tank               ONLINE       0     0     0
      raidz2-0         ONLINE       0     0     0
        concat/vdisk0  ONLINE       0     0     0
        concat/vdisk1  ONLINE       0     0     0
        concat/vdisk2  ONLINE       0     0     0
        concat/vdisk3  ONLINE       0     0     0
        concat/vdisk4  ONLINE       0     0     0

errors: No known data errors
2
  • That seems like a good method until one of the drives breaks. The ZPOOL will keep running (degraded) but how combersome is it to remove the broken drive and get everything back into non-degraded state? I suspect I'd have re-allocate the groups of drives entirely? I could of course replace the broken disk with a bigger one, and use only a part of it, but won't the setup become either a complicated mess or very inefficient after a few disk failures?
    – Zak
    Sep 22, 2021 at 12:06
  • @Zak repairing a degraded pool is a piece of cake, thanks mostly to the design of ZFS. The only extra steps are to destroy the vdisk before replacing the failed drive, and to re-create it afterward. If you're worried about a lack of suitable spare drives available, then I'd suggest building that into your original design, to ensure that you've got a spare 2TB and a spare 384G available. That, or just accept that in case of failure, you'll need to go shop eBay or Craigslist to find some suitable replacements. I wouldn't mess with trying to juggle drives around in a live pool.
    – Jim L.
    Sep 22, 2021 at 18:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.