1

For about a year I had a debian server running with data on dm-crypt/ luks-encrypted hdds underneath zfs (pool name zdata), 8 hdds in a mirrored mode, so 4 mirrors with 2 hdds each. A zpool status from the healthy times:

# zpool status  
pool: zdata  
state: ONLINE  
scrub: scrub completed after (blahblah)  
config:  
NAME        STATE     READ WRITE CKSUM  
zdata          ONLINE       0     0     0  
mirror-0       ONLINE       0     0     0  
 dm-name-n8_2  ONLINE       0     0     0  
 dm-name-n8_3  ONLINE       0     0     0  
mirror-1       ONLINE       0     0     0  
 dm-name-n8_0  ONLINE       0     0     0  
 dm-name-n8_1  ONLINE       0     0     0  
mirror-2       ONLINE       0     0     0  
 dm-name-n8_4  ONLINE       0     0     0  
 dm-name-n8_5  ONLINE       0     0     0  
mirror-3       ONLINE       0     0     0  
 dm-name-n8_7  ONLINE       0     0     0  
 dm-name-n8_6  ONLINE       0     0     0  
errors: No known data errors  

(Of course, no backup of this 34TB raw data. I can give up on almost all data, but there are some files I consider precious to me.)
Two weeks ago after upgrading to stable I noticed several read, write and checksum errors beginning with dm-name-n8_2. A scrub didn't do it, because zfs began a resilvering, but never ended it, as it began over and over again. That was the moment, where also dm-name-n8_3, the other hdd in this mirror, showed same kind of errors and I began to realize, that both hhds respectively their controllers were dying.
I think, than I did silly things, not everything I can comprehend at the moment (zpool history shows nothing). Among others I initiated a replace with the "better" disks. In the meantime I worked on a defect home appliance which caused a power outage, and after booting up I wasn't able to import the pool again. The bad disks are now dd'ed and substituted. Eventually I cannot import the pool again:

# zpool import -f
pool: zdata
id: 1343310357846896221
state: UNAVAIL
status: One or more devices were being resilvered.
action: The pool cannot be imported due to damaged devices or data.
config:

zdata             UNAVAIL  missing device
mirror-0        DEGRADED
 dm-name-n8_2  UNAVAIL
 dm-name-n8_3  ONLINE
mirror-1       ONLINE
 n8_0          ONLINE
 n8_1          ONLINE
mirror-2       DEGRADED
 dm-name-n8_4  ONLINE
 dm-name-n8_5  UNAVAIL
mirror-3       ONLINE
 n8_7          ONLINE
 dm-name-n8_6  ONLINE

# zpool import zdata -f
cannot import 'zdata': one or more devices is currently unavailable

Samesame with zpool import zdata -fnF
Note the changed device names.
The pool has at least one device online in each mirror, why does it complain about a missing device?
How do I get to know, which device is missing? Are there ways to get the pool online again and how do I do it?
(I read the oracle docs and the chapter on zfs at the freebsd handbook, learned some things, but didn't find an answer to my questions.)
The output from zdb is a bit lengthy, it can be seen here.

  • Try using zpool status -P, it will show you the unavailable device absolute paths. – CMCDragonkai Aug 11 '17 at 12:01
  • @CMCDragonkai: (cannot format sufficiently here, sorry) thanks for help, but na, that doesn't work: # zpool status -P no pools available – manes Aug 11 '17 at 16:15
  • Try using with sudo. – CMCDragonkai Aug 12 '17 at 9:38
  • @CMCDragonkaiYepp, I did all this as root. zfs and zpool commands need root. – manes Aug 15 '17 at 19:04
1

Eventually I found a solution for the mess: I saw differing statusmessages dependent on which disks I took to bring the pool up again. I made several attempts to import the pool in a degraded state with various combinations of four corresponding disks, and in the end I got this:

NAME                       STATE     READ WRITE CKSUM
zdata                      DEGRADED     0     0     0
  mirror-0                 DEGRADED     0     0     0
    dm-name-n8_2           ONLINE       0     0     0  (resilvering)
    11141007683912581709   UNAVAIL      0     0     0  was     /dev/disk/by-id/dm-name-n8_3
  mirror-1                 DEGRADED     0     0     0
    16620393607066428577   FAULTED      0     0     0  was     /dev/disk/by-id/dm-name-n8_0
    dm-name-n8_0           ONLINE       0     0     0
  mirror-2                 DEGRADED     0     0     0
    replacing-0            DEGRADED    85     0     0
      1051730541091272780  FAULTED      0     0     0  was     /dev/disk/by-id/dm-name-n8_4
      dm-name-n8_6         ONLINE       0     0    85  (resilvering)
dm-name-n8_4           ONLINE       0     0     0  (resilvering)
  mirror-3                 DEGRADED     0     0     0
    dm-name-n8_5           ONLINE       0     0     0
    13833275275194605312   FAULTED      0     0     0  was     /dev/disk/by-id/dm-name-n8_6

which enabled me to scrape almost all data from the damaged disks. The loss was about ~0.1% (134 of >70000) files.

From my understanding, zfs stores configuration data at each device of that pool, and keeps them synchronized. Maybe the power outage damaged this, or the dying disks?

Anyway, I am happy again. Thanks for reading and helping!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.