I'm new to the ZFS filesystem. I was using a DrivePool in windows, but I wanted something that would correct errors on the fly as well as create and maintain snapshots. I was using BTRFS at one time; however ZFS was included with Ubuntu 16.04 and I've heard that ZFS is more stable than BTRFS in raid-z2 so I figured, why not.

I did the move about three weeks ago. I took 10 4TB drives and put them into a raid-z2. It took about 18 hours from backups to raid. I didn't relize that drive order was importaint. I took one drive out and placed it inside the case since I have 14 ports, but only 12 slots. This apparently killed my z2 without telling me. I wondered why my array was running at 90MB/s but my job kept me from investigating it further until tonight.

zpool status

! I had one drive in FAULTED status and one drive in UNAVAIL. Both drives SMART checks out fine. Since it's been about two weeks the data on the drives is crap. So I OFFLINE the disks.

 root@core:/home/kayot# zpool status
 pool: storage
 state: DEGRADED
 status: One or more devices has been taken offline by the administrator.
    Sufficient replicas exist for the pool to continue functioning in a
    degraded state.
 action: Online the device using 'zpool online' or replace the device with
    'zpool replace'.
 scan: scrub in progress since Tue Feb 21 20:15:16 2017
 151G scanned out of 24.8T at 401M/s, 17h55m to go
 0 repaired, 0.60% done

    NAME                      STATE     READ WRITE CKSUM
    storage                   DEGRADED     0     0     0
      raidz2-0                DEGRADED     0     0     0
        sda                   ONLINE       0     0     0
        sdb                   ONLINE       0     0     0
        sdc                   ONLINE       0     0     0
        sdd                   ONLINE       0     0     0
        sde                   ONLINE       0     0     0
        sdf                   ONLINE       0     0     0
        sdg                   ONLINE       0     0     0
        sdh                   ONLINE       0     0     0
        11017248821379080310  OFFLINE      0     0     0  was /dev/sdk1
        2056055545907538536   OFFLINE      0     0     0  was /dev/sdl1

I have full backups so I'm not nervous or anything. So I figure, I'll replace the disks with the same disks. SDK is still on the system but it's not in use right?

root@core:/home/kayot# zpool replace -f storage 11017248821379080310 /dev/sdk
invalid vdev specification
the following errors must be manually repaired:
/dev/sdk1 is part of active pool 'storage'

That's weird. It openly told me the drive was falted and I OFFLINE'd it. I cleared the partitions. I used Parted and removed the partitions. I also created a new partition GPT.

root@core:/home/kayot# zpool replace storage /dev/sdk
cannot replace /dev/sdk with /dev/sdk: /dev/sdk is busy
root@core:/home/kayot# zpool replace storage /dev/sdk
invalid vdev specification
use '-f' to override the following errors:
/dev/sdk1 is part of active pool 'storage'

Same error. How does it know it's still the same drive?

So I DD the first 10GB of the drive. I then used Parted and created a new partition table. Same error... So then I tried to cross add the drives, I'll use the other drive to replace the first drive etc.

Same error different drive.


I used labelclear as well. Same result.

What do I have to do to get these drives back into service? Nuke the whole array?

So, what am I doing wrong and how would I fix it. I have backups, but I'm thinking of this as a learning experience and if I can fix it instead of replace it, then that would be ideal.

Here is my ZBD Trace

root@core:/home/kayot# zdb
    version: 5000
    name: 'storage'
    state: 0
    txg: 1080550
    pool_guid: 9888777651596013552
    errata: 0
    hostname: 'core'
    vdev_children: 1
        type: 'root'
        id: 0
        guid: 9888777651596013552
            type: 'raidz'
            id: 0
            guid: 6154429632473251558
            nparity: 2
            metaslab_array: 34
            metaslab_shift: 38
            ashift: 12
            asize: 40007384432640
            is_log: 0
            create_txg: 4
                type: 'disk'
                id: 0
                guid: 788698524796940048
                path: '/dev/sda1'
                whole_disk: 1
                DTL: 170
                create_txg: 4
                type: 'disk'
                id: 1
                guid: 13851319173204376663
                path: '/dev/sdb1'
                whole_disk: 1
                DTL: 169
                create_txg: 4
                type: 'disk'
                id: 2
                guid: 2340725902255763627
                path: '/dev/sdc1'
                whole_disk: 1
                DTL: 168
                create_txg: 4
                type: 'disk'
                id: 3
                guid: 17362585713374487716
                path: '/dev/sdd1'
                whole_disk: 1
                DTL: 167
                create_txg: 4
                type: 'disk'
                id: 4
                guid: 2150600564749895821
                path: '/dev/sde1'
                whole_disk: 1
                DTL: 166
                create_txg: 4
                type: 'disk'
                id: 5
                guid: 3210726972626380545
                path: '/dev/sdf1'
                whole_disk: 1
                DTL: 165
                create_txg: 4
                type: 'disk'
                id: 6
                guid: 8795069986910347514
                path: '/dev/sdg1'
                whole_disk: 1
                DTL: 164
                create_txg: 4
                type: 'disk'
                id: 7
                guid: 6469389792976906412
                path: '/dev/sdh1'
                whole_disk: 1
                DTL: 163
                create_txg: 4
                type: 'disk'
                id: 8
                guid: 11017248821379080310
                path: '/dev/sdk1'
                whole_disk: 1
                not_present: 1
                DTL: 160
                create_txg: 4
                offline: 1
                type: 'disk'
                id: 9
                guid: 2056055545907538536
                path: '/dev/sdl1'
                whole_disk: 1
                not_present: 1
                DTL: 159
                create_txg: 4
                offline: 1
  • Just a comment... btrieve file system works great for databases in my past experience when you rebuild it from time to time to fix corruption. I'm not so sure I would trust it with my directory / file tree. Commented Feb 22, 2017 at 2:57
  • I'm using ZFS for my archive so I'm not worried. I just figured that ZFS was more stable(tm).
    – Kayot
    Commented Feb 22, 2017 at 2:58
  • Why did you not just what the message said: action: Online the device using 'zpool online'? zpool replaceis used when you have new disks that were not part of the zpool before. When unplugging a disk, the zpool will be degraded, but when plugging the disk on again it is recognized as part of the pool again, thus ` /dev/sdk1 is part of active pool 'storage'`. And what do you mean with "drive order was important"?
    – ridgy
    Commented Feb 22, 2017 at 14:13
  • When I online the disk it goes back into FAULTED status. The other disk goes back into UNAVAIL status. The order mattered because the UNAVAIL disk is the one I moved.
    – Kayot
    Commented Feb 22, 2017 at 19:18
  • Did you try zpool replace -f storage /dev/sdk as the error message suggested?
    – ridgy
    Commented Feb 24, 2017 at 20:52

2 Answers 2


In order to use again the same drive in ZFS, if you are sure the disk is not faulty, doing DD to the first 10 GB of the drive is a good start, but you need to do it also at the end of the drive. But there is no need to DD 10 GB, I believe the first and last MB is enough.

I solved it this way:

dd bs=512 if=/dev/zero of=/dev/sdk count=2048 seek=$(($(blockdev --getsz /dev/sdk) - 2048))
dd bs=512 if=/dev/zero of=/dev/sdk count=2048 

Then just add the "new" disk back to the ZFS pool. No need to labelclear, scrub, wipe the disk. DD is all you need. You also should not partition the disk after DD. ZFS will not accept any previously partitioned disk.

Then just zpool replace <pool-name> <old-device> <new-device>

normally is the disk-by-id and is the device listed in lsblk.

  • Great! Thanks, this worked, when wipefs -a /dev/sdk did not.
    – Alek
    Commented Apr 20, 2020 at 10:53

ZFS provides built-in functionality for clearing metadata from a device: zpool-labelclear(8)

zpool labelclear [-f] device

Somewhat confusingly, device is the actual device the ZFS metadata sits on - for example, when using full block devices (created with zfs create ... /dev/sda ...), it will be the first partition (/dev/sda1), not the entire drive.

This allows a disk to be removed and re-added cleanly, as if it was a new drive:

# zpool status
  pool: neo
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Jun 16 03:45:28 2024
    5.16T / 44.9T scanned at 2.64G/s, 1.31T / 44.0T issued at 686M/s
    167G resilvered, 2.97% done, 18:09:21 to go

    NAME                                                   STATE     READ WRITE CKSUM
    neo                                                    DEGRADED     0     0     0
      raidz1-0                                             DEGRADED     0     0     0
        scsi-SATA_HGST_HUH728080AL_VLGYAX8Z                ONLINE       0     0     0
        scsi-SATA_HGST_HUH728080AL_VLH5BP7Z                ONLINE       0     0     0
        scsi-SATA_HGST_HUH728080AL_VLHSCA6Y                ONLINE       0     0     0
        replacing-3                                        DEGRADED     0     0     0
          disk//by-id/scsi-SATA_ST8000NM017B-2TJ_WWZEQ5HR  FAULTED      0     0     0  corrupted data
          scsi-SATA_ST8000NM017B-2TJ_WWZEQ5HR              ONLINE       0     0     0  (resilvering)
        scsi-SATA_ST8000NM000A-2KE_WRD0KF47                ONLINE       0     0     0
        scsi-SATA_ST8000NM000A-2KE_WSDGNP3Q                ONLINE       0     0     0
        scsi-SATA_ST8000NM000A-2KE_WSD2WB21                ONLINE       0     0     0
        scsi-SATA_ST8000NM000A-2KE_WSDH2K18                ONLINE       0     0     0

errors: No known data errors

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .