Openindiana oi_151a7. zpool with 15x 1TB raidz3 + 2x 8 disk 2TB raidz2.
Many disk operations such as zpool status, creating a new iscsi volume (using sbdadm) etc hangs. Using iostat (-xne) I can determine that two block devices have errors, one is with high probability a cdrom, floppy or such, and the other is a 2TB disk (of one of the raidz2 vdevs the zpool consist of).
iostat -xne
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 1344 22 1366 c10t32d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 52 385 437 c9t5000C50057CEF65Fd0
The problem with "c9t5000C50057CEF65Fd0" is that it's not listed in 'format'. 'format' further lists all 16 2TB disks, while iostat lists 17. One of the 2TB disks was replaced 2-3 months ago so Im thinking it's possible that iostat still think it's present although its not. Let me point out that I know I have 16 2TB disks physically in the host, and 16 (as 2x8 disk raidz2) in vdevs.
This is a node for backup, so while a reboot is definitely possible it would be very convenient to be able to investigate it further.
Iostat and format in it's entirety:
extended device statistics ---- errors ---
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b s/w h/w trn tot device
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 69 0 69 c5t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 c4t0d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 c4t1d0
5.0 0.0 80.1 0.0 0.0 0.0 0.0 3.4 0 2 0 0 0 0 c4t2d0
7.0 4.0 112.1 96.1 0.0 0.0 0.0 2.7 0 3 0 0 0 0 c4t3d0
13.0 5.0 208.1 184.1 0.0 0.1 0.0 3.0 0 5 0 0 0 0 c4t4d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 c7t50000393D8CAD353d0
143.6 121.1 531.8 681.2 0.0 2.2 0.0 8.3 0 63 0 0 0 0 c7t50000393D8CAD313d0
144.1 132.6 516.6 682.2 0.0 2.4 0.0 8.6 0 69 0 0 0 0 c7t50000393D8CAD243d0
140.1 121.6 614.6 681.9 0.0 3.0 0.0 11.5 0 74 0 0 0 0 c7t50000393D8CAD217d0
142.6 136.6 568.6 678.4 0.0 2.4 0.0 8.5 0 67 0 0 0 0 c7t50000393D8CAD2A7d0
143.1 149.1 539.1 680.9 0.0 2.3 0.0 7.9 0 69 0 0 0 0 c7t50000393D8CAD32Bd0
150.1 118.1 571.1 683.2 0.0 2.7 0.0 10.0 0 71 0 0 0 0 c7t50000393D8CAD29Bd0
143.1 111.6 502.6 679.4 0.0 2.2 0.0 8.7 0 62 0 0 0 0 c7t50000393D8CAD1FBd0
146.1 156.1 539.1 679.2 0.0 2.4 0.0 7.8 0 69 0 0 0 0 c 7t50000393D8CAD24Bd0
146.1 136.6 533.8 680.7 0.0 2.4 0.0 8.5 0 67 0 0 0 0 c7t50000393D8CAD25Bd0
143.1 129.6 529.1 679.9 0.0 2.1 0.0 7.8 0 60 0 0 0 0 c7t50000393D8CABD2Fd0
149.1 123.6 511.6 682.9 0.0 2.5 0.0 9.3 0 73 0 0 0 0 c7t50000393D8CAD35Fd0
142.6 140.6 540.6 680.4 0.0 2.4 0.0 8.3 0 69 0 0 0 0 c7t50000393D8CAD3BFd0
143.6 112.6 555.4 683.4 0.0 2.8 0.0 10.8 0 70 0 0 0 0 c7t50000393D8CAD34Fd0
140.1 143.6 514.1 680.9 0.0 2.5 0.0 8.9 0 71 0 0 0 0 c7t50000393D8CAD33Fd0
138.6 137.6 624.9 682.4 0.0 2.3 0.0 8.4 0 70 0 0 0 0 c7t50000393D8CAD36Fd0
123.1 227.1 494.3 1304.1 0.0 1.3 0.0 3.8 0 35 0 0 0 0 c9t5000C50056114317d0
54.0 122.6 336.2 426.5 0.0 0.3 0.0 1.5 0 8 0 0 0 0 c9t5000C500561161F3d0
51.0 104.1 319.7 427.3 0.0 0.5 0.0 3.2 0 12 0 0 0 0 c9t5000C5005611763Fd0
46.5 101.1 145.3 428.5 0.0 0.3 0.0 2.1 0 9 0 0 0 0 c9t5000C50055FFF9EBd0
38.5 128.6 198.6 427.0 0.0 0.3 0.0 1.6 0 9 0 0 0 0 c9t5000C50055FC6FD3d0
42.5 98.1 168.4 430.0 0.0 0.3 0.0 2.0 0 10 0 0 0 0 c9t5000C50055FFC403d0
53.5 143.6 336.5 423.3 0.0 0.3 0.0 1.5 0 9 0 0 0 0 c9t5000C5005611431Fd0
52.0 143.6 330.2 419.5 0.0 0.3 0.0 1.4 0 8 0 0 0 0 c9t5000C50056000BCBd0
36.0 148.6 270.4 426.5 0.0 0.2 0.0 1.2 0 7 0 0 0 0 c9t5000C50055FC7BF7d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 0 0 0 c4t5d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 1344 22 1366 c10t32d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 1 10 11 c10t33d0
108.6 257.2 642.4 1297.8 0.0 1.0 0.0 2.7 0 29 0 0 0 0 c9t5000C50057CE4F2Bd0
116.1 238.7 620.1 1295.3 0.0 1.1 0.0 3.2 0 34 0 0 0 0 c9t5000C50057CF51D7d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0 52 385 437 c9t5000C50057CEF65Fd0
126.1 261.2 600.4 1294.3 0.0 1.0 0.0 2.5 0 32 0 0 0 0 c9t5000C50057CE4ED7d0
119.6 238.2 653.7 1289.8 0.0 1.0 0.0 2.9 0 32 0 0 0 0 c9t5000C50057CE50A7d0
112.1 220.6 484.6 1300.8 0.0 1.0 0.0 2.9 0 30 0 0 0 0 c9t5000C50057CEF10Bd0
109.1 257.2 572.6 1294.6 0.0 1.1 0.0 3.0 0 31 0 0 0 0 c9t5000C50057CE37DFd0
121.1 242.2 600.6 1293.8 0.0 1.1 0.0 3.0 0 33 0 0 0 0 c9t5000C50057CE4A6Bd0
0. c4t0d0 <HP-LOGICALVOLUME-2.74 cyl 17841 alt 2 hd 255 sec 63>
/pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@0,0
1. c4t1d0 <HP-LOGICALVOLUME-2.74 cyl 17841 alt 2 hd 255 sec 63>
/pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@1,0
2. c4t2d0 <HP-LOGICALVOLUME-2.74 cyl 17841 alt 2 hd 255 sec 63>
/pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@2,0
3. c4t3d0 <HP-LOGICAL VOLUME-2.74-68.33GB>
/pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@3,0
4. c4t4d0 <HP-LOGICAL VOLUME-2.74-68.33GB>
/pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@4,0
5. c4t5d0 <HP-LOGICAL VOLUME-2.74-68.33GB>
/pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@5,0
6. c7t50000393D8CABD2Fd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
/pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cabd2f,0
7. c7t50000393D8CAD1FBd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
/pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad1fb,0
8. c7t50000393D8CAD2A7d0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
/pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad2a7,0
9. c7t50000393D8CAD3BFd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
/pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad3bf,0
10. c7t50000393D8CAD24Bd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
/pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad24b,0
11. c7t50000393D8CAD25Bd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
/pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad25b,0
12. c7t50000393D8CAD29Bd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
/pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad29b,0
13. c7t50000393D8CAD32Bd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
/pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad32b,0
14. c7t50000393D8CAD33Fd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
/pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad33f,0
15. c7t50000393D8CAD34Fd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
/pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad34f,0
16. c7t50000393D8CAD35Fd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
/pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad35f,0
17. c7t50000393D8CAD36Fd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
/pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad36f,0
18. c7t50000393D8CAD217d0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
/pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad217,0
19. c7t50000393D8CAD243d0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
/pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad243,0
20. c7t50000393D8CAD313d0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
/pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad313,0
21. c7t50000393D8CAD353d0 <TOSHIBA-MK1001TRKB-0105 cyl 60798 alt 2 hd 255 sec 126>
/pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad353,0
22. c9t5000C50055FC6FD3d0 <SEAGATE-ST32000645SS-0004-1.82TB>
/scsi_vhci/disk@g5000c50055fc6fd3
23. c9t5000C50055FC7BF7d0 <SEAGATE-ST32000645SS-0004-1.82TB>
/scsi_vhci/disk@g5000c50055fc7bf7
24. c9t5000C50055FFC403d0 <SEAGATE-ST32000645SS-0004-1.82TB>
/scsi_vhci/disk@g5000c50055ffc403
25. c9t5000C50055FFF9EBd0 <SEAGATE-ST32000645SS-0004-1.82TB>
/scsi_vhci/disk@g5000c50055fff9eb
26. c9t5000C50057CE4A6Bd0 <SEAGATE-ST2000NM0023-0003-1.82TB>
/scsi_vhci/disk@g5000c50057ce4a6b
27. c9t5000C50057CE4ED7d0 <SEAGATE-ST2000NM0023-0003-1.82TB>
/scsi_vhci/disk@g5000c50057ce4ed7
28. c9t5000C50057CE4F2Bd0 <SEAGATE-ST2000NM0023-0003-1.82TB>
/scsi_vhci/disk@g5000c50057ce4f2b
29. c9t5000C50057CE37DFd0 <SEAGATE-ST2000NM0023-0003-1.82TB>
/scsi_vhci/disk@g5000c50057ce37df
30. c9t5000C50057CE50A7d0 <SEAGATE-ST2000NM0023-0003-1.82TB>
/scsi_vhci/disk@g5000c50057ce50a7
31. c9t5000C50057CEF10Bd0 <SEAGATE-ST2000NM0023-0003-1.82TB>
/scsi_vhci/disk@g5000c50057cef10b
32. c9t5000C50057CF51D7d0 <SEAGATE-ST2000NM0023-0003-1.82TB>
/scsi_vhci/disk@g5000c50057cf51d7
33. c9t5000C50056000BCBd0 <SEAGATE-ST32000645SS-0004-1.82TB>
/scsi_vhci/disk@g5000c50056000bcb
34. c9t5000C500561161F3d0 <SEAGATE-ST32000645SS-0004-1.82TB>
/scsi_vhci/disk@g5000c500561161f3
35. c9t5000C5005611763Fd0 <SEAGATE-ST32000645SS-0004-1.82TB>
/scsi_vhci/disk@g5000c5005611763f
36. c9t5000C5005611431Fd0 <SEAGATE-ST32000645SS-0004-1.82TB>
/scsi_vhci/disk@g5000c5005611431f
37. c9t5000C50056114317d0 <SEAGATE-ST32000645SS-0004-1.82TB>
/scsi_vhci/disk@g5000c50056114317
38. c10t32d0 <drive type unknown>
/iscsi/disk@0000iqn.1984-05.com.dell%3Apowervault.md3200i.6782bcb000311a0a000000004df9adc70001,0
39. c10t33d0 <drive type unknown>
/iscsi/disk@0000iqn.1984-05.com.dell%3Apowervault.md3200i.6782bcb000311a0a000000004df9adc70001,31