In a Solaris system i use iostat -E and kstat to identify faulted disks. My problem is that I don't know if the problem is on disk, on controller, on bus or on the filesystem layer (ZFS).

The key is understanding the attributes:

Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 0 Predictive Failure Analysis: 0 

I have looked them up on the internet but found nothing concludent.


While iostat and kstat are useful, since you've got Solaris with ZFS (so 10 or later), then you can take advantage of the Fault Management Architecture (FMA), which is tied in with zfs and many other parts of the OS.

Look at the output from 'fmdump' to see if there have been any faults. As an example (with repeated output elided):

$ fmdump -u af2a7c20-05f9-ef95-b016-fcfa812c4f81
TIME                 UUID                                 SUNW-MSG-ID EVENT
Aug 15 15:48:16.3800 af2a7c20-05f9-ef95-b016-fcfa812c4f81 ZFS-8000-LR    Diagnosed
Aug 15 15:58:16.2179 af2a7c20-05f9-ef95-b016-fcfa812c4f81 FMD-8000-9L Isolated
Sep 01 10:11:17.6591 af2a7c20-05f9-ef95-b016-fcfa812c4f81 FMD-8000-9L Isolated
Oct 09 21:45:57.0520 af2a7c20-05f9-ef95-b016-fcfa812c4f81 FMD-8000-9L Isolated
Oct 24 17:21:48.9799 af2a7c20-05f9-ef95-b016-fcfa812c4f81 FMD-8000-9L Isolated
Nov 02 12:41:30.4093 af2a7c20-05f9-ef95-b016-fcfa812c4f81 FMD-8000-4M Repaired
Nov 02 12:41:30.4169 af2a7c20-05f9-ef95-b016-fcfa812c4f81 FMD-8000-6U Resolved

and in slightly more detail:

$ fmdump -v -u af2a7c20-05f9-ef95-b016-fcfa812c4f81
TIME                 UUID                                 SUNW-MSG-ID EVENT
Aug 15 15:48:16.3800 af2a7c20-05f9-ef95-b016-fcfa812c4f81 ZFS-8000-LR Diagnosed
100%  fault.fs.zfs.open_failed

    Problem in: zfs://pool=6e954c71737e8931/vdev=b0b1eb6df8896769/pool_name=soundandvision/vdev_name=id1,sd@SATA_____Hitachi_HDS72101______JPS911N11TBURL/a
       Affects: zfs://pool=6e954c71737e8931/vdev=b0b1eb6df8896769/pool_name=soundandvision/vdev_name=id1,sd@SATA_____Hitachi_HDS72101______JPS911N11TBURL/a
           FRU: zfs://pool=6e954c71737e8931/vdev=b0b1eb6df8896769/pool_name=soundandvision/vdev_name=id1,sd@SATA_____Hitachi_HDS72101______JPS911N11TBURL/a
  FRU Location: -

Note that if you've got your disks inside a jbod with SES location information, or if you've got a supported array then you'd be able to see the location of the disk in question, like this:

$ fmdump -v -u 80db2054-3954-49bf-b705-8820542679fc
TIME                 UUID                                 SUNW-MSG-ID EVENT
Nov 09 08:09:10.0909 80db2054-3954-49bf-b705-8820542679fc DISK-8000-6R Diagnosed
  100%  fault.io.scsi.disk.tran.unstable.leaf

        Problem in: hc://:chassis-mfg=Oracle-Corporation:chassis-name=ORACLE-DE2-24C:chassis-part=31883801+1+1:chassis-serial=1330FMT006:fru-mfg=HITACHI:fru-name=H7230AS60SUN3.0T:fru-serial=001229RSEGSD--------YHKSEGSD:fru-part=HITACHI-H7230AS60SUN3.0T:fru-revision=A142:devid=id1,sd@n5000cca01ad3b688/ses-enclosure=0/bay=16/disk=0
           Affects: dev:///:devid=id1,sd@n5000cca01ad3b688//scsi_vhci/disk@g5000cca01ad3b688
               FRU: hc://:chassis-mfg=Oracle-Corporation:chassis-name=ORACLE-DE2-24C:chassis-part=31883801+1+1:chassis-serial=1330FMT006:fru-mfg=HITACHI:fru-name=H7230AS60SUN3.0T:fru-serial=001229RSEGSD--------YHKSEGSD:fru-part=HITACHI-H7230AS60SUN3.0T:fru-revision=A142:devid=id1,sd@n5000cca01ad3b688/ses-enclosure=0/bay=16/disk=0
      FRU Location: /ORACLE-DE2-24C.1330FMT006/Disk-16
  • Yes, FMA can be good. But from my experience and what I saw, sometimes HBA or controller errors are not shown by FMA. So the solution is to know exactly what those errors mean. – MrDan Nov 19 '14 at 15:20
  • If such errors aren't showing up in FMA, then that is a bug and should be followed up by filing a support request with Oracle to get it figured out. The iostat error types basically match up with those found in the various SCSI family of specifications. – James McPherson Nov 20 '14 at 4:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.