I'm trying to understand what could have gone wrong.

for context:
I have a mirrored set with 3 drives and non-ECC memory. Not sure what else to share.

I was under assumption that having a 3-way mirror would keep me relatively secure from incidental corruptions, like failing drives, or that memory corruption would be recoverable (from one of 2 remaining disks).

But I'm probably misunderstanding.
Can anyone explain what could have gone wrong so I can secure myself for the future?

$ sudo zpool status -v
  pool: dozer
 state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: http://zfsonlinux.org/msg/ZFS-8000-8A
  scan: scrub repaired 0 in 8h36m with 1 errors on Sun Jan  6 02:12:32 2019

    NAME                                          STATE     READ WRITE CKSUM
    dozer                                         ONLINE       0     0     1
      mirror-0                                    ONLINE       0     0     2
        ata-WDC_WD40EFRX-68N32N0_WD-WCC7K1ZKZLYK  ONLINE       0     0     2
        ata-WDC_WD40EFRX-68N32N0_WD-WCC7K6VCAZXL  ONLINE       0     0     2
        ata-ST4000DM000-1F2168_S301LW48           ONLINE       0     0     2

errors: Permanent errors have been detected in the following files:

  • Thanks for the acceptance, favour returned, question upvoted! – Fabby Jan 7 at 20:50

You do understand that if any single non-RAIDed drive has a failure, you lose data and the same is true if one of the non-ECC memory chips has a failure: you lose data.

And even if you have:

  • dual processors
  • dual NICs
  • dual hard drives
  • ECC memory
  • High-availability fail-over servers in a different data centre on a different continent

...any kind of uncorrectable error (Bug, human, Electro-Magnetic Pulse, .. ) can still lead to data loss.

And that's why, with all this nifty technology, we still have multiple off-line backups today.

Note: On-line backups / Data replication to a secondary Data Centre can be corrupted too in this way

  • 2
    When on doubt, backup. If still in doubt, backup again. – Rui F Ribeiro Jan 6 at 12:55
  • Thanks! Yea, I do have a backup, just trying to make sure I haven't set it up in a wrong way or something. I'm also willing to accept the fact of having non-ECC, hence backup. – Bartek Chlebek Jan 6 at 13:28
  • Let me get one thing straight, please. If something messed up in, say, ata-ST4000DM000-1F2168_S301LW48, then zfs would recover from one of the other drives, right? But if the corruption happened in memory, then mirror doesn't help anymore. Is that correct? – Bartek Chlebek Jan 6 at 13:28
  • 1
    Correct. @BartekChlebek (For some or other reason, I never replied to the last comment...) – Fabby Jan 13 at 11:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.