zpool status hangs forever

Question

Openindiana oi_151a7. zpool with 15x 1TB raidz3 + 2x 8 disk 2TB raidz2.

Many disk operations such as zpool status, creating a new iscsi volume (using sbdadm) etc hangs. Using iostat (-xne) I can determine that two block devices have errors, one is with high probability a cdrom, floppy or such, and the other is a 2TB disk (of one of the raidz2 vdevs the zpool consist of).

iostat -xne
0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0 1344  22 1366 c10t32d0
0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0  52 385 437 c9t5000C50057CEF65Fd0

The problem with "c9t5000C50057CEF65Fd0" is that it's not listed in 'format'. 'format' further lists all 16 2TB disks, while iostat lists 17. One of the 2TB disks was replaced 2-3 months ago so Im thinking it's possible that iostat still think it's present although its not. Let me point out that I know I have 16 2TB disks physically in the host, and 16 (as 2x8 disk raidz2) in vdevs.

This is a node for backup, so while a reboot is definitely possible it would be very convenient to be able to investigate it further.

Iostat and format in it's entirety:

                        extended device statistics       ---- errors --- 
r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b s/w h/w trn tot device
0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0  69   0  69 c5t0d0
0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0   0   0   0 c4t0d0
0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0   0   0   0 c4t1d0
5.0    0.0   80.1    0.0  0.0  0.0    0.0    3.4   0   2   0   0   0   0 c4t2d0
7.0    4.0  112.1   96.1  0.0  0.0    0.0    2.7   0   3   0   0   0   0 c4t3d0
13.0    5.0  208.1  184.1  0.0  0.1    0.0    3.0   0   5   0   0   0   0 c4t4d0
0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0   0   0   0 c7t50000393D8CAD353d0
143.6  121.1  531.8  681.2  0.0  2.2    0.0    8.3   0  63   0   0   0   0 c7t50000393D8CAD313d0
144.1  132.6  516.6  682.2  0.0  2.4    0.0    8.6   0  69   0   0   0   0 c7t50000393D8CAD243d0
140.1  121.6  614.6  681.9  0.0  3.0    0.0   11.5   0  74   0   0   0   0 c7t50000393D8CAD217d0
142.6  136.6  568.6  678.4  0.0  2.4    0.0    8.5   0  67   0   0   0   0  c7t50000393D8CAD2A7d0
143.1  149.1  539.1  680.9  0.0  2.3    0.0    7.9   0  69   0   0   0   0 c7t50000393D8CAD32Bd0
150.1  118.1  571.1  683.2  0.0  2.7    0.0   10.0   0  71   0   0   0   0 c7t50000393D8CAD29Bd0
143.1  111.6  502.6  679.4  0.0  2.2    0.0    8.7   0  62   0   0   0   0 c7t50000393D8CAD1FBd0
146.1  156.1  539.1  679.2  0.0  2.4    0.0    7.8   0  69   0   0   0   0 c 7t50000393D8CAD24Bd0
146.1  136.6  533.8  680.7  0.0  2.4    0.0    8.5   0  67   0   0   0   0 c7t50000393D8CAD25Bd0
143.1  129.6  529.1  679.9  0.0  2.1    0.0    7.8   0  60   0   0   0   0 c7t50000393D8CABD2Fd0
149.1  123.6  511.6  682.9  0.0  2.5    0.0    9.3   0  73   0   0   0   0 c7t50000393D8CAD35Fd0
142.6  140.6  540.6  680.4  0.0  2.4    0.0    8.3   0  69   0   0   0   0 c7t50000393D8CAD3BFd0
143.6  112.6  555.4  683.4  0.0  2.8    0.0   10.8   0  70   0   0   0   0 c7t50000393D8CAD34Fd0
140.1  143.6  514.1  680.9  0.0  2.5    0.0    8.9   0  71   0   0   0   0 c7t50000393D8CAD33Fd0
138.6  137.6  624.9  682.4  0.0  2.3    0.0    8.4   0  70   0   0   0   0 c7t50000393D8CAD36Fd0
123.1  227.1  494.3 1304.1  0.0  1.3    0.0    3.8   0  35   0   0   0   0 c9t5000C50056114317d0
 54.0  122.6  336.2  426.5  0.0  0.3    0.0    1.5   0   8   0   0   0   0 c9t5000C500561161F3d0
51.0  104.1  319.7  427.3  0.0  0.5    0.0    3.2   0  12   0   0   0   0 c9t5000C5005611763Fd0
46.5  101.1  145.3  428.5  0.0  0.3    0.0    2.1   0   9   0   0   0   0 c9t5000C50055FFF9EBd0
38.5  128.6  198.6  427.0  0.0  0.3    0.0    1.6   0   9   0   0   0   0 c9t5000C50055FC6FD3d0
42.5   98.1  168.4  430.0  0.0  0.3    0.0    2.0   0  10   0   0   0   0 c9t5000C50055FFC403d0
53.5  143.6  336.5  423.3  0.0  0.3    0.0    1.5   0   9   0   0   0   0 c9t5000C5005611431Fd0
52.0  143.6  330.2  419.5  0.0  0.3    0.0    1.4   0   8   0   0   0   0 c9t5000C50056000BCBd0
36.0  148.6  270.4  426.5  0.0  0.2    0.0    1.2   0   7   0   0   0   0 c9t5000C50055FC7BF7d0
0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0   0   0   0 c4t5d0
0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0 1344  22 1366 c10t32d0
0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0   1  10  11 c10t33d0
108.6  257.2  642.4 1297.8  0.0  1.0    0.0    2.7   0  29   0   0   0   0 c9t5000C50057CE4F2Bd0
116.1  238.7  620.1 1295.3  0.0  1.1    0.0    3.2   0  34   0   0   0   0 c9t5000C50057CF51D7d0
0.0    0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0   0  52 385 437 c9t5000C50057CEF65Fd0
126.1  261.2  600.4 1294.3  0.0  1.0    0.0    2.5   0  32   0   0   0   0 c9t5000C50057CE4ED7d0
119.6  238.2  653.7 1289.8  0.0  1.0    0.0    2.9   0  32   0   0   0   0 c9t5000C50057CE50A7d0
112.1  220.6  484.6 1300.8  0.0  1.0    0.0    2.9   0  30   0   0   0   0 c9t5000C50057CEF10Bd0
109.1  257.2  572.6 1294.6  0.0  1.1    0.0    3.0   0  31   0   0   0   0 c9t5000C50057CE37DFd0
121.1  242.2  600.6 1293.8  0.0  1.1    0.0    3.0   0  33   0   0   0   0 c9t5000C50057CE4A6Bd0


   0. c4t0d0 <HP-LOGICALVOLUME-2.74 cyl 17841 alt 2 hd 255 sec 63>
      /pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@0,0
   1. c4t1d0 <HP-LOGICALVOLUME-2.74 cyl 17841 alt 2 hd 255 sec 63>
      /pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@1,0
   2. c4t2d0 <HP-LOGICALVOLUME-2.74 cyl 17841 alt 2 hd 255 sec 63>
      /pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@2,0
   3. c4t3d0 <HP-LOGICAL VOLUME-2.74-68.33GB>
      /pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@3,0
   4. c4t4d0 <HP-LOGICAL VOLUME-2.74-68.33GB>
      /pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@4,0
   5. c4t5d0 <HP-LOGICAL VOLUME-2.74-68.33GB>
      /pci@0,0/pci1002,5a1e@d/pci103c,3245@0/sd@5,0
   6. c7t50000393D8CABD2Fd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cabd2f,0
   7. c7t50000393D8CAD1FBd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad1fb,0
   8. c7t50000393D8CAD2A7d0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad2a7,0
   9. c7t50000393D8CAD3BFd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad3bf,0
  10. c7t50000393D8CAD24Bd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad24b,0
  11. c7t50000393D8CAD25Bd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad25b,0
  12. c7t50000393D8CAD29Bd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad29b,0
  13. c7t50000393D8CAD32Bd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad32b,0
  14. c7t50000393D8CAD33Fd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad33f,0
  15. c7t50000393D8CAD34Fd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad34f,0
  16. c7t50000393D8CAD35Fd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad35f,0
  17. c7t50000393D8CAD36Fd0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad36f,0
  18. c7t50000393D8CAD217d0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad217,0
  19. c7t50000393D8CAD243d0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad243,0
  20. c7t50000393D8CAD313d0 <TOSHIBA-MK1001TRKB-0105-931.51GB>
      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad313,0
  21. c7t50000393D8CAD353d0 <TOSHIBA-MK1001TRKB-0105 cyl 60798 alt 2 hd 255 sec 126>
      /pci@0,0/pci1002,5a1f@b/pci10b5,8616@0/pci10b5,8616@5/pci1000,3080@0/iport@f0/disk@w50000393d8cad353,0
  22. c9t5000C50055FC6FD3d0 <SEAGATE-ST32000645SS-0004-1.82TB>
      /scsi_vhci/disk@g5000c50055fc6fd3
  23. c9t5000C50055FC7BF7d0 <SEAGATE-ST32000645SS-0004-1.82TB>
      /scsi_vhci/disk@g5000c50055fc7bf7
  24. c9t5000C50055FFC403d0 <SEAGATE-ST32000645SS-0004-1.82TB>
      /scsi_vhci/disk@g5000c50055ffc403
  25. c9t5000C50055FFF9EBd0 <SEAGATE-ST32000645SS-0004-1.82TB>
      /scsi_vhci/disk@g5000c50055fff9eb
  26. c9t5000C50057CE4A6Bd0 <SEAGATE-ST2000NM0023-0003-1.82TB>
      /scsi_vhci/disk@g5000c50057ce4a6b
  27. c9t5000C50057CE4ED7d0 <SEAGATE-ST2000NM0023-0003-1.82TB>
      /scsi_vhci/disk@g5000c50057ce4ed7
  28. c9t5000C50057CE4F2Bd0 <SEAGATE-ST2000NM0023-0003-1.82TB>
      /scsi_vhci/disk@g5000c50057ce4f2b
  29. c9t5000C50057CE37DFd0 <SEAGATE-ST2000NM0023-0003-1.82TB>
      /scsi_vhci/disk@g5000c50057ce37df
  30. c9t5000C50057CE50A7d0 <SEAGATE-ST2000NM0023-0003-1.82TB>
      /scsi_vhci/disk@g5000c50057ce50a7
  31. c9t5000C50057CEF10Bd0 <SEAGATE-ST2000NM0023-0003-1.82TB>
      /scsi_vhci/disk@g5000c50057cef10b
  32. c9t5000C50057CF51D7d0 <SEAGATE-ST2000NM0023-0003-1.82TB>
      /scsi_vhci/disk@g5000c50057cf51d7
  33. c9t5000C50056000BCBd0 <SEAGATE-ST32000645SS-0004-1.82TB>
      /scsi_vhci/disk@g5000c50056000bcb
  34. c9t5000C500561161F3d0 <SEAGATE-ST32000645SS-0004-1.82TB>
      /scsi_vhci/disk@g5000c500561161f3
  35. c9t5000C5005611763Fd0 <SEAGATE-ST32000645SS-0004-1.82TB>
      /scsi_vhci/disk@g5000c5005611763f
  36. c9t5000C5005611431Fd0 <SEAGATE-ST32000645SS-0004-1.82TB>
      /scsi_vhci/disk@g5000c5005611431f
  37. c9t5000C50056114317d0 <SEAGATE-ST32000645SS-0004-1.82TB>
      /scsi_vhci/disk@g5000c50056114317
  38. c10t32d0 <drive type unknown>
      /iscsi/disk@0000iqn.1984-05.com.dell%3Apowervault.md3200i.6782bcb000311a0a000000004df9adc70001,0
  39. c10t33d0 <drive type unknown>
      /iscsi/disk@0000iqn.1984-05.com.dell%3Apowervault.md3200i.6782bcb000311a0a000000004df9adc70001,31

Rui F Ribeiro · Accepted Answer · 2019-04-06 01:30:56Z

This is a short answer on a question that is very old and probably already either not an issue any longer or you simply don't care any more. Since I've had the same problem and found a solution for mine I thought I'd answer here.

It's quite simple. add

myIPaddress  myhostname myhostname.FullyQualifiedDomainNameIFIHaveone

In to the /etc/hosts file since for some reason it looks to the resolver library when looking doing a zpool status -v

pool: vmpool03
state: ONLINE
scan: scrub repaired 0 in 0h0m with 0 errors on Fri Dec 16 23:46:02 2016
config:
    NAME                                        STATE     READ WRITE CKSUM
    vmpool03                                    ONLINE       0     0     0
      raidz1-0                                  ONLINE       0     0     0
        scsi-3600508b1001c5189c2e106add2e3b9df  ONLINE       0     0     0
        scsi-3600508b1001c013352baa8831e9b60fb  ONLINE       0     0     0
errors: No known data errors

In my case the difference was before adding the IP and external address 4min till it finished and after, a fraction of a second.

This was done on ZFSonLinux installation with centOS 7.2<

Thanks! I do care still and I've had these kinds of issues from time to time, more often than not when disks or hbas not responding. However name resolving sounds like another possible reason, good that you shared it! — 3molo, Dec 22 '16 at 13:20

Stack Exchange Network

current community

your communities

more stack exchange communities

zpool status hangs forever

1 Answer 1

Your Answer

Not the answer you're looking for? Browse other questions tagged zfs openindiana sasl or ask your own question.

Hot Network Questions

zpool status hangs forever

1 Answer 1

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged zfs openindiana sasl or ask your own question.

Related

Hot Network Questions