Why does ZFS scrub not complete?

Question

First off let's start with what I'm running. This is a home media server running Ubuntu 16.10. I have one pool of mirrored 6 terabyte drives that are about half full. I built the system about a month ago, and it's running great. It uses an SSD as a boot drive and the above-mentioned pool as storage. I have been able to do everything I needed with this pool and all seems great.

Both drives were new when I built the system about a month ago, and I was a little curious about some extra vibrations with one of them. Nothing bad, but the vendor said he would replace it at no charge, so I was planning to run a scrub and pull it out to send it in, running in a degraded state while I wait. There is no non backed up data on it, so I'm not terribly concerned, but obviously, it would be easier to do it this way as opposed to killing the pool and restoring from backup.

All I'm really trying to do at the moment is run the scrub and safely detach the one drive from the mirror. I run

zpool scrub tank

and then immediately run

zpool status

and I can see the scrub happening. I can run an update every few seconds and see it update the status just fine. It runs for about 30 seconds and then status does not show it running anymore. Also, I have never seen anything other than last scrub completed in 0 hours 0 minutes from the status. To me, that means that the scrub is not happening to completion, as should not a scrub take at least several hours with two and a half terabytes of info to go through.

What am I missing?

---

adding requested info:

  pool: Tank
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Sun Feb  5 00:31:42 2017
config:

        NAME        STATE     READ WRITE CKSUM
        Tank        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdb2    ONLINE       0     0     0
            sdc2    ONLINE       0     0     0

errors: No known data errors

I'm trying a scrub again now just to make sure the issue is still current. Here is a status about 20 seconds after I start...

  pool: Tank
 state: ONLINE
  scan: scrub in progress since Fri Feb 10 14:25:12 2017
    62.5M scanned out of 2.97T at 1.08M/s, (scan is slow, no estimated time)
    0 repaired, 0.00% done
config:

    NAME        STATE     READ WRITE CKSUM
    Tank        ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        sdb2    ONLINE       0     0     0
        sdc2    ONLINE       0     0     0

errors: No known data errors

and here it is again after about a minute...

  pool: Tank
 state: ONLINE
  scan: scrub repaired 0 in 0h1m with 0 errors on Fri Feb 10 14:27:01 2017
config:

    NAME        STATE     READ WRITE CKSUM
    Tank        ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        sdb2    ONLINE       0     0     0
        sdc2    ONLINE       0     0     0

errors: No known data errors

---

edit for add'l info on 2/16/17

I'm running out of time to send the "noisy" drive back so I pulled it. I did nothing but unplug it (while the system was up). Everything continues to function properly for the moment, albeit in a DEGRADED state as expected. I guess I'll keep documenting my experience here since I've already started to. Sounds like nobody else is having this problem. I can't seem to find anyone else on the net with the same situation. Lucky me. We will see what happens when I get the replacement drive and resilver. Who knows... maybe the data gods will have mercy on me and simply replacing the drive will force the problem to fix itself. :/ Below is my output after disconnecting the drive.

  pool: Tank
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: scrub repaired 0 in 0h1m with 0 errors on Sun Feb 12 00:24:38 2017
config:

        NAME        STATE     READ WRITE CKSUM
        Tank        DEGRADED     0     0     0
          mirror-0  DEGRADED     0     0     0
            sdb2    ONLINE       0     0     0
            sdc2    UNAVAIL      0     0     0

errors: No known data errors

---

edit for add'l info on 3/29/17

root@NAS:~# zpool status
  pool: Tank
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: resilvered 525M in 0h3m with 0 errors on Wed Mar 29 14:28:46 2017
config:

        NAME        STATE     READ WRITE CKSUM
        Tank        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdb2    ONLINE       0     0     0
            sdc     ONLINE       0     0   732

errors: No known data errors

Maybe another clue to issues? look at the partition of sdc...

root@NAS:/dev# parted --list
Model: ATA Samsung SSD 850 (scsi)
Disk /dev/sda: 250GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End    Size    File system     Name                  Flags
 1      1049kB  538MB  537MB   fat32           EFI System Partition  boot, esp
 2      538MB   233GB  232GB   ext4
 3      233GB   250GB  17.1GB  linux-swap(v1)


Warning: Not all of the space available to /dev/sdb appears to be used, you can
fix the GPT to use all of the space (an extra 7 blocks) or continue with the
current setting?
Fix/Ignore? i
Model: ATA HGST HUH728060AL (scsi)
Disk /dev/sdb: 6001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
 1      2097kB  2150MB  2147MB
 2      2150MB  6001GB  5999GB  zfs


Model: ATA HGST HUH728060AL (scsi)
Disk /dev/sdc: 6001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name                  Flags
 1      1049kB  6001GB  6001GB  zfs          zfs-802af6a53a6d8383
 9      6001GB  6001GB  8389kB

---

edit for add'l info on 4/13/17

Yes, I've been trying to fix this issue for months :/

First off, after exporting/importing drive letters changed, so note that sdb became sdc, and sdc became sdd.

I think I found the issue, and I want to get advice on how to fix it. The issue was finally discovered when I ran "sudo fdisk -l". Below are the pertinent snips...

Disk /dev/sdc: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 7127FE7D-E061-11E6-BD1F-3497F600DDAF

 Device     Start        End       Sectors    Size    Type
/dev/sdc1   4096       4198399     4194304     2G     FreeBSD swap
/dev/sdc2  4198400   11721043967 11716845568  5.5T    FreeBSD ZFS

...

Disk /dev/sdd: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: E799A1D5-F9B7-C843-AB62-AADC9B0A2180

Device        Start           End         Sectors    Size    Type
/dev/sdd1      2048       11721027583   11721025536  5.5T    Solaris /usr & Apple ZFS
/dev/sdd9  11721027584    11721043967      16384      8M     Solaris reserved 1

Notice this: the mirror was originally created in FreeNAS (FreeBSD). The sdc has a 2G swap at the beginning of the drive. sdd was created in Ubuntu, and for whatever reason was given an 8M swap at the end of the drive.

Now, fearing that the issue was a bad drive, I offlined sdd and ran badblocks on it. This wipes all info. The good news is that the drive is fine, no bad blocks. This also resets the partitions to nothing.

I have two choices. 1.) Try to manually match the partitions of sdd to the working drive (sdc). Although I thought zfs was supposed to do this automatically by just doing a zpool replace, so maybe that's a waste of time. 2.) I have the data backed up, so I could wipe both drives and start from scratch, create a new mirror, and let it be a native Ubuntu pool.

Maybe this is overthinking it, but I would think I run a bit more risk destroying and restoring. I'm having to destroy good data that's only backed up on a non-mirrored disk, then rsyncing it back to a newly created pool. FYI, I used rsync to create the backup, and it's on the same pc. I had to stripe 3 drives together with no redundancy to fit all the data on it. I'm also afraid that moving the data around like that with no way to scrub it might give me some corruption and I would never know.

Anyone have any advice? Thanks!

Answer 1

The scrub is not completing properly, possibly due to partition mismatches or other underlying issues from the original FreeNAS setup.

First, check dmesg for errors:

dmesg | grep -i zfs

Then export and re-import pool:

zpool export Tank
zpool import Tank

Make sure that both drives use the same partition scheme. Fix partitions on sdc to match sdb:

parted /dev/sdc -- mklabel gpt
parted /dev/sdc -- mkpart primary 2150MB 100%

Replace drive:

zpool replace Tank <old_drive> <new_drive>

Re-scrub the pool:

zpool scrub Tank

Alternatively, you could create a new pool on the replacement drive and transfer data:

zpool create NewPool mirror /dev/sdd
zfs send -R Tank | zfs receive -F NewPool

Thank you!