I'm new to the ZFS filesystem. I was using a DrivePool in windows, but I wanted something that would correct errors on the fly as well as create and maintain snapshots. I was using BTRFS at one time; however ZFS was included with Ubuntu 16.04 and I've heard that ZFS is more stable than BTRFS in raid-z2 so I figured, why not.
I did the move about three weeks ago. I took 10 4TB drives and put them into a raid-z2. It took about 18 hours from backups to raid. I didn't relize that drive order was importaint. I took one drive out and placed it inside the case since I have 14 ports, but only 12 slots. This apparently killed my z2 without telling me. I wondered why my array was running at 90MB/s but my job kept me from investigating it further until tonight.
! I had one drive in FAULTED status and one drive in UNAVAIL. Both drives SMART checks out fine. Since it's been about two weeks the data on the drives is crap. So I OFFLINE the disks.
root@core:/home/kayot# zpool status pool: storage state: DEGRADED status: One or more devices has been taken offline by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Online the device using 'zpool online' or replace the device with 'zpool replace'. scan: scrub in progress since Tue Feb 21 20:15:16 2017 151G scanned out of 24.8T at 401M/s, 17h55m to go 0 repaired, 0.60% done config: NAME STATE READ WRITE CKSUM storage DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 sda ONLINE 0 0 0 sdb ONLINE 0 0 0 sdc ONLINE 0 0 0 sdd ONLINE 0 0 0 sde ONLINE 0 0 0 sdf ONLINE 0 0 0 sdg ONLINE 0 0 0 sdh ONLINE 0 0 0 11017248821379080310 OFFLINE 0 0 0 was /dev/sdk1 2056055545907538536 OFFLINE 0 0 0 was /dev/sdl1
I have full backups so I'm not nervous or anything. So I figure, I'll replace the disks with the same disks. SDK is still on the system but it's not in use right?
root@core:/home/kayot# zpool replace -f storage 11017248821379080310 /dev/sdk invalid vdev specification the following errors must be manually repaired: /dev/sdk1 is part of active pool 'storage'
That's weird. It openly told me the drive was falted and I OFFLINE'd it. I cleared the partitions. I used Parted and removed the partitions. I also created a new partition GPT.
root@core:/home/kayot# zpool replace storage /dev/sdk cannot replace /dev/sdk with /dev/sdk: /dev/sdk is busy root@core:/home/kayot# zpool replace storage /dev/sdk invalid vdev specification use '-f' to override the following errors: /dev/sdk1 is part of active pool 'storage' root@core:/home/kayot#
Same error. How does it know it's still the same drive?
So I DD the first 10GB of the drive. I then used Parted and created a new partition table. Same error... So then I tried to cross add the drives, I'll use the other drive to replace the first drive etc.
Same error different drive.
I used labelclear as well. Same result.
What do I have to do to get these drives back into service? Nuke the whole array?
So, what am I doing wrong and how would I fix it. I have backups, but I'm thinking of this as a learning experience and if I can fix it instead of replace it, then that would be ideal.
Here is my ZBD Trace
root@core:/home/kayot# zdb storage: version: 5000 name: 'storage' state: 0 txg: 1080550 pool_guid: 9888777651596013552 errata: 0 hostname: 'core' vdev_children: 1 vdev_tree: type: 'root' id: 0 guid: 9888777651596013552 children: type: 'raidz' id: 0 guid: 6154429632473251558 nparity: 2 metaslab_array: 34 metaslab_shift: 38 ashift: 12 asize: 40007384432640 is_log: 0 create_txg: 4 children: type: 'disk' id: 0 guid: 788698524796940048 path: '/dev/sda1' whole_disk: 1 DTL: 170 create_txg: 4 children: type: 'disk' id: 1 guid: 13851319173204376663 path: '/dev/sdb1' whole_disk: 1 DTL: 169 create_txg: 4 children: type: 'disk' id: 2 guid: 2340725902255763627 path: '/dev/sdc1' whole_disk: 1 DTL: 168 create_txg: 4 children: type: 'disk' id: 3 guid: 17362585713374487716 path: '/dev/sdd1' whole_disk: 1 DTL: 167 create_txg: 4 children: type: 'disk' id: 4 guid: 2150600564749895821 path: '/dev/sde1' whole_disk: 1 DTL: 166 create_txg: 4 children: type: 'disk' id: 5 guid: 3210726972626380545 path: '/dev/sdf1' whole_disk: 1 DTL: 165 create_txg: 4 children: type: 'disk' id: 6 guid: 8795069986910347514 path: '/dev/sdg1' whole_disk: 1 DTL: 164 create_txg: 4 children: type: 'disk' id: 7 guid: 6469389792976906412 path: '/dev/sdh1' whole_disk: 1 DTL: 163 create_txg: 4 children: type: 'disk' id: 8 guid: 11017248821379080310 path: '/dev/sdk1' whole_disk: 1 not_present: 1 DTL: 160 create_txg: 4 offline: 1 children: type: 'disk' id: 9 guid: 2056055545907538536 path: '/dev/sdl1' whole_disk: 1 not_present: 1 DTL: 159 create_txg: 4 offline: 1 features_for_read: com.delphix:hole_birth com.delphix:embedded_data