Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a Solaris 11 machine that randomly crashed this morning. After physically restarting the machine, I noticed that all of the drives were marked with a Sense Key: Soft_Error both in dmesg and in /var/adm/messages.

Since all the drives on the machine were tagged with the same Soft Error, does this mean that the HBA is faulty? Anyone have any ideas/suggestions?

root@solaris-machine:/var/log# iostat -E
sd0       Soft Errors: 1 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product:       Revision: SN02 Serial No: 
Size: 500.11GB <500107862016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 1
Illegal Request: 12 Predictive Failure Analysis: 0
sd2       Soft Errors: 1 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product:      Revision: 0004 Serial No:  
Size: 3000.59GB <3000592982016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 1
Illegal Request: 0 Predictive Failure Analysis: 0 
sd4       Soft Errors: 1 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product:      Revision: 0004 Serial No: 
Size: 3000.59GB <3000592982016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 1 
Illegal Request: 0 Predictive Failure Analysis: 0 
sd5       Soft Errors: 1 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product:      Revision: 0004 Serial No: 
Size: 3000.59GB <3000592982016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 1 
Illegal Request: 0 Predictive Failure Analysis: 0




Jan 23 10:45:02 solaris-machine scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk@g5000c5004dfae642 (sd4):
Jan 23 10:45:02 solaris-machine      Error for Command: <undecoded cmd 0xa1>    Error Level: Recovered
Jan 23 10:45:02 solaris-machine scsi: [ID 107833 kern.notice]        Requested Block: 0                         Error Block: 0
Jan 23 10:45:02 solaris-machine scsi: [ID 107833 kern.notice]        Vendor: ATA                                Serial Number:        
Jan 23 10:45:02 solaris-machine scsi: [ID 107833 kern.notice]        Sense Key: Soft_Error
Jan 23 10:45:04 solaris-machine scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk@g5000c5004dfc8db2 (sd2):
Jan 23 10:45:04 solaris-machine      Error for Command: <undecoded cmd 0xa1>    Error Level: Recovered
Jan 23 10:45:04 solaris-machine scsi: [ID 107833 kern.notice]        Requested Block: 0                         Error Block: 0
Jan 23 10:45:04 solaris-machine scsi: [ID 107833 kern.notice]        Vendor: ATA                                Serial Number:        
Jan 23 10:45:04 solaris-machine scsi: [ID 107833 kern.notice]        Sense Key: Soft_Error
Jan 23 10:45:04 solaris-machine scsi: [ID 107833 kern.notice]        ASC: 0x0 (<vendor unique code 0x0>), ASCQ: 0x1d, FRU: 0x0
Jan 23 10:45:04 solaris-machine scsi: [ID 107833 kern.warning] WARNING: /scsi_vhci/disk@g5000c5004dfd4ce3 (sd5):
Jan 23 10:45:04 solaris-machine      Error for Command: <undecoded cmd 0xa1>    Error Level: Recovered
Jan 23 10:45:04 solaris-machine scsi: [ID 107833 kern.notice]        Requested Block: 0                         Error Block: 0
Jan 23 10:45:04 solaris-machine scsi: [ID 107833 kern.notice]        Vendor: ATA                                Serial Number:
Jan 23 10:45:04 solaris-machine scsi: [ID 107833 kern.notice]        Sense Key: Soft_Error
Jan 23 10:45:04 solaris-machine scsi: [ID 107833 kern.notice]        ASC: 0x0 (<vendor unique code 0x0>), ASCQ: 0x1d, FRU: 0x0
Jan 23 10:45:07 solaris-machine scsi: [ID 107833 kern.warning] WARNING: /pci@0,0/pci15d9,664@1f,2/disk@0,0 (sd0):
Jan 23 10:45:07 solaris-machine      Error for Command: <undecoded cmd 0xa1>    Error Level: Recovered
Jan 23 10:45:07 solaris-machine scsi: [ID 107833 kern.notice]        Requested Block: 0                         Error Block: 0
Jan 23 10:45:07 solaris-machine scsi: [ID 107833 kern.notice]        Vendor: ATA                                Serial Number:
Jan 23 10:45:07 solaris-machine scsi: [ID 107833 kern.notice]        Sense Key: Soft_Error
Jan 23 10:45:07 solaris-machine scsi: [ID 107833 kern.notice]        ASC: 0x0 (no additional sense info), ASCQ: 0x0, FRU: 0x0
share|improve this question
    
If you got a panic, you should see the in the /Var/adm/messages the panic string. It should point you into the right direction. Also, check the messages prior to the panic to check if there are messages leading to the root cause of the problem. –  BitsOfNix Jan 24 '13 at 6:49
add comment

3 Answers

Where I look when a Solaris 11 server crashes:

  • dmesg | less
    • have a careful read of this, there are clues here, or if the dmesg buffer has cycled, look at the /var/adm/messages log in play at the time of the crash.
  • fmadm faulty
    • this will show any devices the fault manager has deemed faulty, including disks and hbas, and NIC, etc.
  • fmdump -eV
    • this will show you any errors from the system, recognized by the fault management system, regardless of whether they resulted in a piece of hardware being marked faulty.
  • if the disks in question are part of a zpool, then run zpool scrub on that pool, and then look at zpool status and see if you're getting checksum errors that could indicate a problem with the HBA or disks.
  • make sure that the dump config is set to capture crash dumps, by running dumpadm. the output of this should show you that savecore is enabled, and crashes will be left in /var/crash. If your system left a crashdump in /var/crash from this morning's crash, then you can look at the pstack of the crash, and possibly get some indicators as to root cause.

Those steps should get you closer to finding out why your system crashed.

share|improve this answer
add comment

[No Solaris expert anymore, but know my way around hardware with Linux] If it was one drive, I'd suspect that one is failing. But all of them is extra fishy. Trouble with the controller perhaps?

Most crashes I've seen recently were due to CPU overheating (bad fan, dirt, clogged air circulation), close second hangs due to graphics drivers (particularly nVidia). Some trouble spots in times past were brownouts, or even micro blackouts.

share|improve this answer
add comment

Small numbers of soft errors can be safely ignored. Solaris is just pedantic about recording and reporting everything that other OS'es don't even notice.

Monitor the error count (It gets reset on reboot) and check for patterns.

Also be aware that soft errors on one device have a tendency to cause the count on other drives on the same bus to also increment. You will also have a clear winner though, one drive with many times more errors than the others.

I've never found drive error counts (soft or hard errors) to be indicative of HBA failures, but that may just be me. HBA failures typically show up as other messages, eg mpt errors.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.