6

I have had an ongoing saga with a home data server I have set up, and have switched out almost any parts other than the drives themselves.

Starting by using software RAID in CentOS, I've had one series of 5 drives operating, literally flawlessly, for two years in RAID 0. Totally the most dangerous way to run a RAID. The other five drives, identical and from the same batch as the first 5, have always been in some form of RAID 5 configuration, firstly using software RAID, then later in ZFS after a complete rebuild. This set has always, periodically, after months of bulletproof service, just given up and gone offline in more or less spectacular ways.

The drives have lived in external enclosures connected initially by multiplexed eSATA and now by multiplexed USB3.

At first I thought the issue might be with the cheap enclosure/multiplexer, so I swapped the 5 drives of the RAID 0 and the RAID 5 arrays between the two enclosures I had. The RAID 0 continued flawlessly, the RAID 5 continued to have these periodic blackouts.

I had the first hint that the issue was with a drive in the set, but no single one of the five drives has ever had more issues than any other. So I wondered if maybe RAID 5 had some strange power requirement that was tripping the enclosure, and invested in another enclosure, this time a USB 3 connected box - the USB3 is so much more positive than the eSATA.

So that has worked solidly for six months now, until today. On the terminal I received 5 sequential messages:

WARNING: Your hard drive is failing
Device: /dev/sda [SAT], unable to open device
WARNING: Your hard drive is failing
Device: /dev/sdb [SAT], unable to open device
WARNING: Your hard drive is failing
Device: /dev/sdc [SAT], unable to open device
WARNING: Your hard drive is failing
Device: /dev/sdd [SAT], unable to open device
WARNING: Your hard drive is failing
Device: /dev/sde [SAT], unable to open device

I have eliminated the box, the connection the multiplexer, the PCIe eSATA extension boards, the problem must surely be with the drives, but short of throwing them all away, I can't think how to debug this. When it first happened zpool status showed near equal errors for all drives, and it is curious that they all went out in alphabetical order.

I zpool cleared, it resilvered, everything was good for a while, then it stopped responding. Now spool status literally hangs the terminal and is immune to Ctrl+C.

New information:

/dev/sda-e have spontaneously renamed themselves to /dev/sda1-e1, Since there was no read or write I power cycled the drive box. The devices disappeared then reappeared as expected, but still with the 1 suffixes on their names.

Update: (06/03/2017)

Using the Oracle documentation I tried setting failmode to continue:

zpool set failmode=continue tank

In this mode I continue to periodically get

WARNING: Your hard drive is failing
Device: /dev/sda [SAT], unable to open device

and the drives in the array all accrue write errors :

   NAME                        STATE     READ WRITE CKSUM
    tank                        ONLINE       0    16    59
      raidz1-0                  ONLINE       0    32   118
        ata-WDC_WDC_WD10-68...  ONLINE       0    14     0
        ata-WDC_WDC_WD10-68...  ONLINE       0    12     0
        sda                     ONLINE       0    12     0
        ata-WDC_WDC_WD10-68...  ONLINE       0    12     0
        ata-WDC_WDC_WD10-68...  ONLINE       0    14     0

errors: 67 data errors, use '-v' for a list

however at this point at least zpool stays alive and does not hang a terminal indefinitely or hang other pools.

It is interesting that only the writes are accruing errors on all of the drives, and in very equal numbers.

Improve this question
1
  • Though you are right about the dangers of "RAID" 0 (not (R)edundant so not really RAID), the one advantage is that each disk gets less wear. So, each disk of a 5-disk stripe after 2 years would have seen the same amount of wear as a single disk would have in ~7 months. So in that sense, depending on how heavily they are used they aren't that old. My one point of confusion is what zpool configuration you are using with the 5 disks: zraid1? mirror? I think ckujau's answer may be right but I wanted to at least give it a good thinking through.
    – airhuff
    Jan 14, 2017 at 4:54

1 Answer 1

Reset to default
5

As the message is generated by smartdnotify and the system is really having trouble accessing the device, I'd recommend to investigate the drive issues first, as this looks like a hardware problem.

And there's nothing zfs can do about this. Once the faulty hard drive (or cable or controller) has been replaced, zfs may be able to restore the pool again.

Improve this answer
4
  • ckujau I am wondering how exactly to do this, to investigate. The links you sent relate to unplugging an external drive. These drives were never disconnected.
    – J Collins
    Jan 3, 2017 at 20:49
  • As this has nothing to do with ZFS or Unix, the question is a bit misplaced here, I'd recommend Super User instead. Or ask your local hardware dealer to lend you a drive to see if the problem goes away. If not, replace the cables, disk controller, mainboard, memory (possibly again, and in that order). Good luck.
    – ckujau
    Jan 3, 2017 at 20:59
  • The specific question around ZFS is how to deal with the hang up of any zpool commands, or any program that attempts to access the disk. Changing hardware is an expensive task and hoes nowhere to identifying which drive is faulty other than by accident or months of validation time.
    – J Collins
    Jan 5, 2017 at 15:47
  • On review of your answer, (only just noticing the link text colour!) this is a good answer.
    – J Collins
    Dec 12, 2017 at 17:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .