0

My pool:

❯ zpool status
pool: wdblack
state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Mon Oct  5 13:35:14 2020
2.84T scanned at 586M/s, 852G issued at 172M/s, 9.52T total
66.5M resilvered, 8.74% done, 0 days 14:44:07 to go
remove: Removal of vdev 0 copied 5.99T in 15h17m, completed on Sat Jun  6 03:15:41 2020
29.5M memory used for removed device mappings
config:

NAME                                     STATE     READ WRITE CKSUM
wdblack                                  ONLINE       0     0     0
wwn-0x50014ee26390e982                 ONLINE       0     0    82
wwn-0x5000cca27ec99833-part1           ONLINE       0     0  264K  (resilvering)
ata-WDC_WD80EZAZ-11TDBA0_JEH52XAN      ONLINE       0     0     0
ata-GOODRAM_C100_FF180744082600124110  ONLINE       0     0     0
sdg                                    ONLINE       0     0     0

The -part1 has been in resilvering loop for a week now. It starts itself every time.

Zfs features:

NAME     PROPERTY              VALUE                  SOURCE
wdblack  type                  filesystem             -
wdblack  creation              Sat Mar  9 10:54 2019  -
wdblack  used                  9.56T                  -
wdblack  available             10.5T                  -
wdblack  referenced            96K                    -
wdblack  compressratio         1.00x                  -
wdblack  mounted               yes                    -
wdblack  quota                 none                   default
wdblack  reservation           none                   default
wdblack  recordsize            128K                   local
wdblack  mountpoint            /home/agilob/disk      local
wdblack  sharenfs              off                    default
wdblack  checksum              on                     default
wdblack  compression           off                    default
wdblack  atime                 off                    local
wdblack  devices               on                     default
wdblack  exec                  on                     default
wdblack  setuid                on                     default
wdblack  readonly              off                    default
wdblack  zoned                 off                    default
wdblack  snapdir               hidden                 default
wdblack  aclinherit            restricted             default
wdblack  createtxg             1                      -
wdblack  canmount              on                     default
wdblack  xattr                 on                     default
wdblack  copies                1                      default
wdblack  version               5                      -
wdblack  utf8only              off                    -
wdblack  normalization         none                   -
wdblack  casesensitivity       sensitive              -
wdblack  vscan                 off                    default
wdblack  nbmand                off                    default
wdblack  sharesmb              off                    default
wdblack  refquota              none                   default
wdblack  refreservation        none                   default
wdblack  guid                  5626685650647801653    -
wdblack  primarycache          all                    default
wdblack  secondarycache        all                    default
wdblack  usedbysnapshots       0B                     -
wdblack  usedbydataset         96K                    -
wdblack  usedbychildren        9.56T                  -
wdblack  usedbyrefreservation  0B                     -
wdblack  logbias               latency                default
wdblack  objsetid              51                     -
wdblack  dedup                 off                    default
wdblack  mlslabel              none                   default
wdblack  sync                  standard               default
wdblack  dnodesize             legacy                 default
wdblack  refcompressratio      1.00x                  -
wdblack  written               96K                    -
wdblack  logicalused           9.56T                  -
wdblack  logicalreferenced     42K                    -
wdblack  volmode               default                default
wdblack  filesystem_limit      none                   default
wdblack  snapshot_limit        none                   default
wdblack  filesystem_count      none                   default
wdblack  snapshot_count        none                   default
wdblack  snapdev               hidden                 default
wdblack  acltype               off                    default
wdblack  context               none                   default
wdblack  fscontext             none                   default
wdblack  defcontext            none                   default
wdblack  rootcontext           none                   default
wdblack  relatime              off                    local
wdblack  redundant_metadata    all                    default
wdblack  overlay               off                    default
wdblack  encryption            off                    default
wdblack  keylocation           none                   default
wdblack  keyformat             none                   default
wdblack  pbkdf2iters           0                      default
wdblack  special_small_blocks  0                      default

I'm unable to detach any device, even those that aren't resilvering

❯ sudo zpool detach wdblack /dev/sdg
cannot detach /dev/sdg: only applicable to mirror and replacing vdevs
❯ sudo zpool detach wdblack wwn-0x5000cca27ec99833-part1
cannot detach wwn-0x5000cca27ec99833-part1: only applicable to mirror and replacing vdevs

Can't remove then either

❯ sudo zpool remove wdblack wwn-0x5000cca27ec99833-part1
cannot remove wwn-0x5000cca27ec99833-part1: Pool busy; removal may already be in progress
❯ sudo zpool remove wdblack sdg
cannot remove sdg: invalid config; all top-level vdevs must have the same sector size and not be raidz.

If I remember correctly, resilvering has started itself after starting scrubing, which can't be changed:

❯ sudo zpool scrub -s wdblack
cannot cancel scrubbing wdblack: currently resilvering
❯ sudo zpool scrub wdblack
cannot scrub wdblack: currently resilvering

zpool status wdblack reports there are files damaged, but the number changes "randomly" it's usually between 7 (most often) and over 3000000, right now it's errors: 259134 data errors, use '-v' for a list

over 300000 is number of my all files on the pool!

Right now it reports over 259k errors, but when I sudo zpool status wdblack -v it can only print 7 files, not 259k.

I'm using archlinux, at the time of writing

Linux 5.8.12-arch1-1 #1 SMP PREEMPT Sat, 26 Sep 2020 21:42:58 +0000 x86_64 GNU/Linux
❯ zfs version
zfs-0.8.4-1
zfs-kmod-0.8.4-1

Edit:

Resilver starts instantly after finishing

Oct  6 2020 18:41:03.798523850 sysevent.fs.zfs.resilver_finish
version = 0x0
class = "sysevent.fs.zfs.resilver_finish"
pool = "wdblack"
pool_guid = 0x509d876228a22ecc
pool_state = 0x0
pool_context = 0x0
time = 0x5f7cac2f 0x2f9881ca 
eid = 0x27e4e

Oct  6 2020 18:41:03.798523850 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "wdblack"
pool_guid = 0x509d876228a22ecc
pool_state = 0x0
pool_context = 0x0
history_hostname = "slave"
history_internal_str = "errors=159477"
history_internal_name = "starting deferred resilver"
history_txg = 0x7246f3
history_time = 0x5f7cac2f
time = 0x5f7cac2f 0x2f9881ca 
eid = 0x27e4f

Oct  6 2020 18:41:09.005195427 sysevent.fs.zfs.resilver_start
version = 0x0
class = "sysevent.fs.zfs.resilver_start"
pool = "wdblack"
pool_guid = 0x509d876228a22ecc
pool_state = 0x0
pool_context = 0x0
time = 0x5f7cac35 0x4f46a3
eid = 0x27e50

1 Answer 1

1

This happened to me once, as soon as one resilvering loop finished I ran a scrub of the pools. It didn't find any errors but the resilvering stopped looping. After that I was able to remove and replace drives

Update:

Well, well, well will you look at that. 3 hours after your last comment they fixed it. It has been in progress since May 4 and they released 0.8.5 with the fix. https://github.com/openzfs/zfs/pull/10291

Unfortunately you'll have to build ZFS for this fix.

Or find a trusted source with a package for version 0.8.5.

You can get the ZFS source from here: https://github.com/openzfs/zfs/releases/tag/zfs-0.8.5

It's your lucky day! :)

5
  • This was my thinking too, so far I wasn't able to catch that moment when resilvering finishes. ETA is super unreliable, every time I run zpool status ETA changes by several seconds, every few minutes it changes by a few minutes. Might have to write a script for that... While writing this comment ETA changed by 3 minutes 20 seconds.
    – agilob
    Oct 5, 2020 at 18:44
  • please see edit :(
    – agilob
    Oct 6, 2020 at 17:54
  • you gotta look at the remaining GB, I don't use the ETA
    – Tek
    Oct 6, 2020 at 19:09
  • It doesn't matter, resilver started the same millisecond as finished.
    – agilob
    Oct 6, 2020 at 19:26
  • 1
    Yes, it was a bug in zfs. Timing of me finding it, dealing for days, and and they fixing it... I confirmed the fix.
    – agilob
    Oct 9, 2020 at 7:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .