3

I have an older (2-3 years) NAS running FreeNAS. I figured its time to upgrade some of the hard drives. Here's the current status:

    NAME        STATE     READ WRITE CKSUM                                  
    Pool1       ONLINE       0     0     0                                  
      raidz1    ONLINE       0     0     0                                  
        ada3p2  ONLINE       0     0     0                                  
        ada4p2  ONLINE       0     0     0                                  
        ada5p2  ONLINE       0     0     0                                  
      mirror    ONLINE       0     0     0                                  
        ada1p2  ONLINE       0     0     0                                  
        ada2p2  ONLINE       0     0     0 

The drives in RaidZ1 are all 1GB and will stay.

What's the best way to add 2 or 3 more 1GB hard drives to RaidZ1 and completely remove the mirrored drives, which are 500GB only, without:

  • Copying the data to a temporary holding spot
  • Recreating the pool
  • Copying it back

The pool is about 50% full, so I'm not even sure there's data on the mirrored drives.

How to check or where to begin?

1
  • 1
    Surely you mean 1 TB instead of "1GB", right? Mar 30, 2018 at 0:38

3 Answers 3

1

The only way to do it is as you suggest: copy all the datasets to a new pool and rebuild this one. zfs does not support extending raidz1's by adding drives to it, and it doesn't support removing top-level vdevs either (which is what 'mirror' is).

2
1

Short answer:

What's the best way to add ... more ... drives to RaidZ1 and completely remove the mirrored drives, ... without:

  • Copying the data to a temporary holding spot
  • Recreating the pool
  • Copying it back

You can't. Full stop. Given your configuration, you will not be able to remove the mirrored drives without creating a new pool and copying to it. Especially given the time of your post, ZFS does not support removal of VDEVs from a pool. For readers who encounter this issue in more recent times, as of this writing, zpool remove does not support removing VDEVs from a pool that contains one or more raidz VDEVs. zpool remove does support removal of concatenated disks or concatenated mirrors.

Long answer:

You have a somewhat unusual configuration of a raidz1 vdev concatenated with a mirror vdev. You won't be able to remove the mirror vdev from the pool, except by creating a new pool, copying the data from the old pool to the new pool, and then destroying the old pool.

But you can certainly expand the capacity of your current pool, so long as you don't change its vdev structure.

So instead of adding more drives, you can switch to larger drives. It's relatively painless, although it will entail several iterations of resilvering the pool onto each new drive, one at a time. But ZFS is more efficient about resilvering than traditional RAID is, so depending on how full your pool is, it might not be so bad.

The key is that this MUST be done one disk at a time. Start by making sure you have a full and tested backup of your pool.

Here's a test sandbox set up with three 1G drives, devices /dev/md0 through /dev/md2, with a GPT partition table on each:

# gpart show md{0..2}
=>     40  2097072  md0  GPT  (1.0G)
       40      200    1  linux-swap  (100K)
      240  2096872    2  freebsd-zfs  (1.0G)

=>     40  2097072  md1  GPT  (1.0G)
       40      200    1  linux-swap  (100K)
      240  2096872    2  freebsd-zfs  (1.0G)

=>     40  2097072  md2  GPT  (1.0G)
       40      200    1  linux-swap  (100K)
      240  2096872    2  freebsd-zfs  (1.0G)

Out of those drives, I've built a raidz1 pool:

# zpool create tank raidz1 /dev/md{0..2}p2
# zpool status tank; echo; zpool list tank
  pool: tank
 state: ONLINE
config:

    NAME        STATE     READ WRITE CKSUM
    tank        ONLINE       0     0     0
      raidz1-0  ONLINE       0     0     0
        md0p2   ONLINE       0     0     0
        md1p2   ONLINE       0     0     0
        md2p2   ONLINE       0     0     0

errors: No known data errors

NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH
tank  2.75G   196K  2.75G        -         -     0%     0%  1.00x    ONLINE

Lovely. Now, suppose I have three new drives, /dev/md3, md4 and md5. But instead of being only 1G drives, these are 10G drives. They also are set up with GPT partition tables and are ready to go:

# gpart show md{3..5}
=>      40  20971440  md3  GPT  (10G)
        40       200    1  linux-swap  (100K)
       240  20971240    2  freebsd-zfs  (10G)

=>      40  20971440  md4  GPT  (10G)
        40       200    1  linux-swap  (100K)
       240  20971240    2  freebsd-zfs  (10G)

=>      40  20971440  md5  GPT  (10G)
        40       200    1  linux-swap  (100K)
       240  20971240    2  freebsd-zfs  (10G)

Because the pool is redundant, I can carefully replace one drive in the raidz1 array with one larger drive, then wait for the pool to resilver, and then move on the replace the next drive, one at a time. I'll do the first one the long way, by offline-ing the drive, inspecting the array to see that it's still operating, and then replacing the small drive with the large one:

# zpool offline tank md0p2
# zpool status tank
  pool: tank
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
    Sufficient replicas exist for the pool to continue functioning in a
    degraded state.
action: Online the device using 'zpool online' or replace the device with
    'zpool replace'.
config:

    NAME        STATE     READ WRITE CKSUM
    tank        DEGRADED     0     0     0
      raidz1-0  DEGRADED     0     0     0
        md0p2   OFFLINE      0     0     0
        md1p2   ONLINE       0     0     0
        md2p2   ONLINE       0     0     0

errors: No known data errors

Now I can replace the 1G /dev/md0p2 partition with the 10G partition /dev/md3p2. In practice, the zpool offline step above isn't strictly necessary. It's possible to go directly to this zpool replace command:

# zpool replace tank md0p2 md3p2
# zpool status tank; echo; zpool list tank
  pool: tank
 state: ONLINE
  scan: resilvered 128K in 00:00:02 with 0 errors on Wed Oct 13 13:46:23 2021
config:

    NAME        STATE     READ WRITE CKSUM
    tank        ONLINE       0     0     0
      raidz1-0  ONLINE       0     0     0
        md3p2   ONLINE       0     0     0
        md1p2   ONLINE       0     0     0
        md2p2   ONLINE       0     0     0

errors: No known data errors

NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH
tank  2.75G   265K  2.75G        -         -     0%     0%  1.00x    ONLINE

Notice that the resilvering is complete. This is crucial. If you have a significant amount of data stored in your pool, you will have to wait, checking zpool status tank ocassionally, until the resilvering finishes.

Once the resilver is complete, we can proceed to replace the second disk:

# zpool replace tank md1p2 md4p2

And again, we must wait until zpool status says the resilvering has finished:

# zpool status tank
  pool: tank
 state: ONLINE
  scan: resilvered 134K in 00:00:03 with 0 errors on Wed Oct 13 13:54:17 2021
config:

    NAME        STATE     READ WRITE CKSUM
    tank        ONLINE       0     0     0
      raidz1-0  ONLINE       0     0     0
        md3p2   ONLINE       0     0     0
        md4p2   ONLINE       0     0     0
        md2p2   ONLINE       0     0     0

errors: No known data errors

Now that the resilver has finished, we'll replace the third small disk with the larger replacement:

# zpool replace tank md2p2 md5p2
# zpool status tank; echo; zpool list tank
  pool: tank
 state: ONLINE
  scan: resilvered 146K in 00:00:03 with 0 errors on Wed Oct 13 14:55:56 2021
config:

    NAME        STATE     READ WRITE CKSUM
    tank        ONLINE       0     0     0
      raidz1-0  ONLINE       0     0     0
        md3p2   ONLINE       0     0     0
        md4p2   ONLINE       0     0     0
        md5p2   ONLINE       0     0     0

errors: No known data errors

NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH
tank  2.75G   319K  2.75G        -         -     0%     0%  1.00x    ONLINE

So now the old, small disks md0 md1 and md2 are no longer in the pool. The pool is running on the larger disks md3 md4 and md5. But there's no additional space, because we haven't told zpool to expand the raidz1 array. That's easy to do:

# zpool online -e tank md5p2
# zpool status tank; echo; zpool list tank
  pool: tank
 state: ONLINE
  scan: resilvered 146K in 00:00:03 with 0 errors on Wed Oct 13 14:55:56 2021
config:

    NAME        STATE     READ WRITE CKSUM
    tank        ONLINE       0     0     0
      raidz1-0  ONLINE       0     0     0
        md3p2   ONLINE       0     0     0
        md4p2   ONLINE       0     0     0
        md5p2   ONLINE       0     0     0

errors: No known data errors

NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH
tank  29.8G   394K  29.7G        -         -     0%     0%  1.00x    ONLINE

Now instead of a 3G pool we've got a 30G pool, accomplished by carefully replacing one disk at a time, and then expanding the array.

0

First before making any changes, you may want to consider kicking off a zpool scrub to ensure everything is healthy.

Going off memory (so you'll want to confirm), I believe the option you'll want is zpool add to be able to grow your existing raidz pool.

To easily move the data around, you can use zfs send|receive, along with using snapshots for incremental send|receives.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .