ZFS Ubuntu 16.04 Replace drive with itself

Question

I'm new to the ZFS filesystem. I was using a DrivePool in windows, but I wanted something that would correct errors on the fly as well as create and maintain snapshots. I was using BTRFS at one time; however ZFS was included with Ubuntu 16.04 and I've heard that ZFS is more stable than BTRFS in raid-z2 so I figured, why not.

I did the move about three weeks ago. I took 10 4TB drives and put them into a raid-z2. It took about 18 hours from backups to raid. I didn't relize that drive order was importaint. I took one drive out and placed it inside the case since I have 14 ports, but only 12 slots. This apparently killed my z2 without telling me. I wondered why my array was running at 90MB/s but my job kept me from investigating it further until tonight.

zpool status

! I had one drive in FAULTED status and one drive in UNAVAIL. Both drives SMART checks out fine. Since it's been about two weeks the data on the drives is crap. So I OFFLINE the disks.

 root@core:/home/kayot# zpool status
 pool: storage
 state: DEGRADED
 status: One or more devices has been taken offline by the administrator.
    Sufficient replicas exist for the pool to continue functioning in a
    degraded state.
 action: Online the device using 'zpool online' or replace the device with
    'zpool replace'.
 scan: scrub in progress since Tue Feb 21 20:15:16 2017
 151G scanned out of 24.8T at 401M/s, 17h55m to go
 0 repaired, 0.60% done
 config:

    NAME                      STATE     READ WRITE CKSUM
    storage                   DEGRADED     0     0     0
      raidz2-0                DEGRADED     0     0     0
        sda                   ONLINE       0     0     0
        sdb                   ONLINE       0     0     0
        sdc                   ONLINE       0     0     0
        sdd                   ONLINE       0     0     0
        sde                   ONLINE       0     0     0
        sdf                   ONLINE       0     0     0
        sdg                   ONLINE       0     0     0
        sdh                   ONLINE       0     0     0
        11017248821379080310  OFFLINE      0     0     0  was /dev/sdk1
        2056055545907538536   OFFLINE      0     0     0  was /dev/sdl1

I have full backups so I'm not nervous or anything. So I figure, I'll replace the disks with the same disks. SDK is still on the system but it's not in use right?

root@core:/home/kayot# zpool replace -f storage 11017248821379080310 /dev/sdk
invalid vdev specification
the following errors must be manually repaired:
/dev/sdk1 is part of active pool 'storage'

That's weird. It openly told me the drive was falted and I OFFLINE'd it. I cleared the partitions. I used Parted and removed the partitions. I also created a new partition GPT.

root@core:/home/kayot# zpool replace storage /dev/sdk
cannot replace /dev/sdk with /dev/sdk: /dev/sdk is busy
root@core:/home/kayot# zpool replace storage /dev/sdk
invalid vdev specification
use '-f' to override the following errors:
/dev/sdk1 is part of active pool 'storage'
root@core:/home/kayot#

Same error. How does it know it's still the same drive?

So I DD the first 10GB of the drive. I then used Parted and created a new partition table. Same error... So then I tried to cross add the drives, I'll use the other drive to replace the first drive etc.

Same error different drive.

WHY?

I used labelclear as well. Same result.

What do I have to do to get these drives back into service? Nuke the whole array?

So, what am I doing wrong and how would I fix it. I have backups, but I'm thinking of this as a learning experience and if I can fix it instead of replace it, then that would be ideal.

Here is my ZBD Trace

root@core:/home/kayot# zdb
storage:
    version: 5000
    name: 'storage'
    state: 0
    txg: 1080550
    pool_guid: 9888777651596013552
    errata: 0
    hostname: 'core'
    vdev_children: 1
    vdev_tree:
        type: 'root'
        id: 0
        guid: 9888777651596013552
        children[0]:
            type: 'raidz'
            id: 0
            guid: 6154429632473251558
            nparity: 2
            metaslab_array: 34
            metaslab_shift: 38
            ashift: 12
            asize: 40007384432640
            is_log: 0
            create_txg: 4
            children[0]:
                type: 'disk'
                id: 0
                guid: 788698524796940048
                path: '/dev/sda1'
                whole_disk: 1
                DTL: 170
                create_txg: 4
            children[1]:
                type: 'disk'
                id: 1
                guid: 13851319173204376663
                path: '/dev/sdb1'
                whole_disk: 1
                DTL: 169
                create_txg: 4
            children[2]:
                type: 'disk'
                id: 2
                guid: 2340725902255763627
                path: '/dev/sdc1'
                whole_disk: 1
                DTL: 168
                create_txg: 4
            children[3]:
                type: 'disk'
                id: 3
                guid: 17362585713374487716
                path: '/dev/sdd1'
                whole_disk: 1
                DTL: 167
                create_txg: 4
            children[4]:
                type: 'disk'
                id: 4
                guid: 2150600564749895821
                path: '/dev/sde1'
                whole_disk: 1
                DTL: 166
                create_txg: 4
            children[5]:
                type: 'disk'
                id: 5
                guid: 3210726972626380545
                path: '/dev/sdf1'
                whole_disk: 1
                DTL: 165
                create_txg: 4
            children[6]:
                type: 'disk'
                id: 6
                guid: 8795069986910347514
                path: '/dev/sdg1'
                whole_disk: 1
                DTL: 164
                create_txg: 4
            children[7]:
                type: 'disk'
                id: 7
                guid: 6469389792976906412
                path: '/dev/sdh1'
                whole_disk: 1
                DTL: 163
                create_txg: 4
            children[8]:
                type: 'disk'
                id: 8
                guid: 11017248821379080310
                path: '/dev/sdk1'
                whole_disk: 1
                not_present: 1
                DTL: 160
                create_txg: 4
                offline: 1
            children[9]:
                type: 'disk'
                id: 9
                guid: 2056055545907538536
                path: '/dev/sdl1'
                whole_disk: 1
                not_present: 1
                DTL: 159
                create_txg: 4
                offline: 1
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data

Answer 1

In order to use again the same drive in ZFS, if you are sure the disk is not faulty, doing DD to the first 10 GB of the drive is a good start, but you need to do it also at the end of the drive. But there is no need to DD 10 GB, I believe the first and last MB is enough.

I solved it this way:

dd bs=512 if=/dev/zero of=/dev/sdk count=2048 seek=$(($(blockdev --getsz /dev/sdk) - 2048))
dd bs=512 if=/dev/zero of=/dev/sdk count=2048 

Then just add the "new" disk back to the ZFS pool. No need to labelclear, scrub, wipe the disk. DD is all you need. You also should not partition the disk after DD. ZFS will not accept any previously partitioned disk.

Then just zpool replace <pool-name> <old-device> <new-device>

normally is the disk-by-id and is the device listed in lsblk.