I'm new to the ZFS filesystem. I was using a DrivePool in windows, but I wanted something that would correct errors on the fly as well as create and maintain snapshots. I was using BTRFS at one time; however ZFS was included with Ubuntu 16.04 and I've heard that ZFS is more stable than BTRFS in raid-z2 so I figured, why not.
I did the move about three weeks ago. I took 10 4TB drives and put them into a raid-z2. It took about 18 hours from backups to raid. I didn't relize that drive order was importaint. I took one drive out and placed it inside the case since I have 14 ports, but only 12 slots. This apparently killed my z2 without telling me. I wondered why my array was running at 90MB/s but my job kept me from investigating it further until tonight.
zpool status
! I had one drive in FAULTED status and one drive in UNAVAIL. Both drives SMART checks out fine. Since it's been about two weeks the data on the drives is crap. So I OFFLINE the disks.
root@core:/home/kayot# zpool status
pool: storage
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: scrub in progress since Tue Feb 21 20:15:16 2017
151G scanned out of 24.8T at 401M/s, 17h55m to go
0 repaired, 0.60% done
config:
NAME STATE READ WRITE CKSUM
storage DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
sda ONLINE 0 0 0
sdb ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
sdg ONLINE 0 0 0
sdh ONLINE 0 0 0
11017248821379080310 OFFLINE 0 0 0 was /dev/sdk1
2056055545907538536 OFFLINE 0 0 0 was /dev/sdl1
I have full backups so I'm not nervous or anything. So I figure, I'll replace the disks with the same disks. SDK is still on the system but it's not in use right?
root@core:/home/kayot# zpool replace -f storage 11017248821379080310 /dev/sdk
invalid vdev specification
the following errors must be manually repaired:
/dev/sdk1 is part of active pool 'storage'
That's weird. It openly told me the drive was falted and I OFFLINE'd it. I cleared the partitions. I used Parted and removed the partitions. I also created a new partition GPT.
root@core:/home/kayot# zpool replace storage /dev/sdk
cannot replace /dev/sdk with /dev/sdk: /dev/sdk is busy
root@core:/home/kayot# zpool replace storage /dev/sdk
invalid vdev specification
use '-f' to override the following errors:
/dev/sdk1 is part of active pool 'storage'
root@core:/home/kayot#
Same error. How does it know it's still the same drive?
So I DD the first 10GB of the drive. I then used Parted and created a new partition table. Same error... So then I tried to cross add the drives, I'll use the other drive to replace the first drive etc.
Same error different drive.
WHY?
I used labelclear as well. Same result.
What do I have to do to get these drives back into service? Nuke the whole array?
So, what am I doing wrong and how would I fix it. I have backups, but I'm thinking of this as a learning experience and if I can fix it instead of replace it, then that would be ideal.
Here is my ZBD Trace
root@core:/home/kayot# zdb
storage:
version: 5000
name: 'storage'
state: 0
txg: 1080550
pool_guid: 9888777651596013552
errata: 0
hostname: 'core'
vdev_children: 1
vdev_tree:
type: 'root'
id: 0
guid: 9888777651596013552
children[0]:
type: 'raidz'
id: 0
guid: 6154429632473251558
nparity: 2
metaslab_array: 34
metaslab_shift: 38
ashift: 12
asize: 40007384432640
is_log: 0
create_txg: 4
children[0]:
type: 'disk'
id: 0
guid: 788698524796940048
path: '/dev/sda1'
whole_disk: 1
DTL: 170
create_txg: 4
children[1]:
type: 'disk'
id: 1
guid: 13851319173204376663
path: '/dev/sdb1'
whole_disk: 1
DTL: 169
create_txg: 4
children[2]:
type: 'disk'
id: 2
guid: 2340725902255763627
path: '/dev/sdc1'
whole_disk: 1
DTL: 168
create_txg: 4
children[3]:
type: 'disk'
id: 3
guid: 17362585713374487716
path: '/dev/sdd1'
whole_disk: 1
DTL: 167
create_txg: 4
children[4]:
type: 'disk'
id: 4
guid: 2150600564749895821
path: '/dev/sde1'
whole_disk: 1
DTL: 166
create_txg: 4
children[5]:
type: 'disk'
id: 5
guid: 3210726972626380545
path: '/dev/sdf1'
whole_disk: 1
DTL: 165
create_txg: 4
children[6]:
type: 'disk'
id: 6
guid: 8795069986910347514
path: '/dev/sdg1'
whole_disk: 1
DTL: 164
create_txg: 4
children[7]:
type: 'disk'
id: 7
guid: 6469389792976906412
path: '/dev/sdh1'
whole_disk: 1
DTL: 163
create_txg: 4
children[8]:
type: 'disk'
id: 8
guid: 11017248821379080310
path: '/dev/sdk1'
whole_disk: 1
not_present: 1
DTL: 160
create_txg: 4
offline: 1
children[9]:
type: 'disk'
id: 9
guid: 2056055545907538536
path: '/dev/sdl1'
whole_disk: 1
not_present: 1
DTL: 159
create_txg: 4
offline: 1
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
action: Online the device using 'zpool online'
?zpool replace
is used when you have new disks that were not part of thezpool
before. When unplugging a disk, the zpool will be degraded, but when plugging the disk on again it is recognized as part of the pool again, thus ` /dev/sdk1 is part of active pool 'storage'`. And what do you mean with "drive order was important"?zpool replace -f storage /dev/sdk
as the error message suggested?