DEGRADED zpool can't be imported with I/O error

Question

I'm running a personal server at home with a CentOS 7 OS and a 12TB zpool. It's been running for a couple of years and yesterday I noticed some problems so I went in to have a look.

At first it seemed like one of my drives had failed, with zpool import giving the following results:

pool: media
id: 1363376331138686016
state: DEGRADED
status: One or more devices contains corrupted data.
action: The pool can be imported despite missing or damaged devices.  
The
fault tolerance of the pool may be compromised if imported.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
 config:

media                                DEGRADED
  raidz1-0                           DEGRADED
    ata-ST3000DM001-1ER166_W500G55Q  ONLINE
    ata-ST3000DM001-1CH166_Z1F278KB  UNAVAIL
    sdc                              ONLINE
    sde                              ONLINE
    sdf                              ONLINE

This looks ok, however I can't seem to import the pool directly.

Running zpool import media gives me:

cannot import 'media': I/O error
Destroy and re-create the pool from
a backup source.

I've looked around to figure out the problem but everything I've found has given me nothing.

Some other things I've tried:

• zpool import -fFX: zpool could not be imported
• zpool import -fFV: zpool imported with FAULTED status
• zpool status: no pools available

Is anyone able to point me in the right direction? I'm not sure what my next course of diagnosis should be.

Answer 1

Your best bet is to destroy the pool, recreate it with a replacement for the failed drive ata-ST3000DM001-1CH166_Z1F278KB, and then restore from backup.

If that's not an option (it should be - neither ZFS nor RAID are a substitute for backups! nor were they ever intended to be), then you could try taking the zpool offline until you have a replacement for ata-ST3000DM001-1CH166_Z1F278KB ready to be installed.

When you have acquired a replacement drive, try to import the pool in DEGRADED state and immediately replace the failed drive with the good new drive.

BTW, https://serverfault.com/questions/548568/zfs-recover-from-faulted-pool-state may have some useful information for you.

Answer 2

What kind of problems did you notice that made you go and look at the volume? Your zpool command listed shows the drive as unavailable. Can you see it otherwise, or is it dead?

It appears there are sufficient drives in the raidz pool where you should be able to import it.

Normally first with a zpool import -nF to determine if the pool could be imported by discarding the last few transactions.

Sometimes a zpool clear along with a zpool scrub can help. You might also be able to try a zpool replace on the suspect/bad/failing drive with a new drive.

And when using large SATA drives in a home fileserver, it's best not to get too greedy. There are tons of various chains on this whether you're using ZFS, another kind of software raid, as well as even using "enterprise" hardware raid, so I'm not going to go into great detail.

Drives fail. It's going to happen. It is after all just spinning rust. Even if you have a setup where multiple drives could fail, what's to say another drive won't fail during the rebuild to a replacement?--hello multi-terabyte drives. For this reason a lot of people are starting to use 3-way mirrors. Luckily drives aren't as expensive as they used to be.

With ZFS, you could also try increasing the amount of copies (zfs set copies=<# copies> <pool>) to increase the amount of copies for a file to limit possible data loss.