First off let's start with what I'm running. This is a home media server running Ubuntu 16.10. I have one pool of mirrored 6 terabyte drives that are about half full. I built the system about a month ago, and it's running great. It uses an SSD as a boot drive and the above-mentioned pool as storage. I have been able to do everything I needed with this pool and all seems great.
Both drives were new when I built the system about a month ago, and I was a little curious about some extra vibrations with one of them. Nothing bad, but the vendor said he would replace it at no charge, so I was planning to run a scrub and pull it out to send it in, running in a degraded state while I wait. There is no non backed up data on it, so I'm not terribly concerned, but obviously, it would be easier to do it this way as opposed to killing the pool and restoring from backup.
All I'm really trying to do at the moment is run the scrub and safely detach the one drive from the mirror. I run
zpool scrub tank
and then immediately run
and I can see the scrub happening. I can run an update every few seconds and see it update the status just fine. It runs for about 30 seconds and then status does not show it running anymore. Also, I have never seen anything other than last scrub completed in 0 hours 0 minutes from the status. To me, that means that the scrub is not happening to completion, as should not a scrub take at least several hours with two and a half terabytes of info to go through.
What am I missing?
adding requested info:
pool: Tank state: ONLINE scan: scrub repaired 0 in 0h0m with 0 errors on Sun Feb 5 00:31:42 2017 config: NAME STATE READ WRITE CKSUM Tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sdb2 ONLINE 0 0 0 sdc2 ONLINE 0 0 0 errors: No known data errors
I'm trying a scrub again now just to make sure the issue is still current. Here is a status about 20 seconds after I start...
pool: Tank state: ONLINE scan: scrub in progress since Fri Feb 10 14:25:12 2017 62.5M scanned out of 2.97T at 1.08M/s, (scan is slow, no estimated time) 0 repaired, 0.00% done config: NAME STATE READ WRITE CKSUM Tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sdb2 ONLINE 0 0 0 sdc2 ONLINE 0 0 0 errors: No known data errors
and here it is again after about a minute...
pool: Tank state: ONLINE scan: scrub repaired 0 in 0h1m with 0 errors on Fri Feb 10 14:27:01 2017 config: NAME STATE READ WRITE CKSUM Tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sdb2 ONLINE 0 0 0 sdc2 ONLINE 0 0 0 errors: No known data errors
edit for add'l info on 2/16/17
I'm running out of time to send the "noisy" drive back so I pulled it. I did nothing but unplug it (while the system was up). Everything continues to function properly for the moment, albeit in a DEGRADED state as expected. I guess I'll keep documenting my experience here since I've already started to. Sounds like nobody else is having this problem. I can't seem to find anyone else on the net with the same situation. Lucky me. We will see what happens when I get the replacement drive and resilver. Who knows... maybe the data gods will have mercy on me and simply replacing the drive will force the problem to fix itself. :/ Below is my output after disconnecting the drive.
pool: Tank state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: http://zfsonlinux.org/msg/ZFS-8000-4J scan: scrub repaired 0 in 0h1m with 0 errors on Sun Feb 12 00:24:38 2017 config: NAME STATE READ WRITE CKSUM Tank DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 sdb2 ONLINE 0 0 0 sdc2 UNAVAIL 0 0 0 errors: No known data errors
edit for add'l info on 3/29/17
root@NAS:~# zpool status pool: Tank state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://zfsonlinux.org/msg/ZFS-8000-9P scan: resilvered 525M in 0h3m with 0 errors on Wed Mar 29 14:28:46 2017 config: NAME STATE READ WRITE CKSUM Tank ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sdb2 ONLINE 0 0 0 sdc ONLINE 0 0 732 errors: No known data errors
Maybe another clue to issues? look at the partition of sdc...
root@NAS:/dev# parted --list Model: ATA Samsung SSD 850 (scsi) Disk /dev/sda: 250GB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049kB 538MB 537MB fat32 EFI System Partition boot, esp 2 538MB 233GB 232GB ext4 3 233GB 250GB 17.1GB linux-swap(v1) Warning: Not all of the space available to /dev/sdb appears to be used, you can fix the GPT to use all of the space (an extra 7 blocks) or continue with the current setting? Fix/Ignore? i Model: ATA HGST HUH728060AL (scsi) Disk /dev/sdb: 6001GB Sector size (logical/physical): 512B/4096B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 2097kB 2150MB 2147MB 2 2150MB 6001GB 5999GB zfs Model: ATA HGST HUH728060AL (scsi) Disk /dev/sdc: 6001GB Sector size (logical/physical): 512B/4096B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049kB 6001GB 6001GB zfs zfs-802af6a53a6d8383 9 6001GB 6001GB 8389kB
edit for add'l info on 4/13/17
Yes, I've been trying to fix this issue for months :/
First off, after exporting/importing drive letters changed, so note that sdb became sdc, and sdc became sdd.
I think I found the issue, and I want to get advice on how to fix it. The issue was finally discovered when I ran "sudo fdisk -l". Below are the pertinent snips...
Disk /dev/sdc: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: gpt Disk identifier: 7127FE7D-E061-11E6-BD1F-3497F600DDAF Device Start End Sectors Size Type /dev/sdc1 4096 4198399 4194304 2G FreeBSD swap /dev/sdc2 4198400 11721043967 11716845568 5.5T FreeBSD ZFS ... Disk /dev/sdd: 5.5 TiB, 6001175126016 bytes, 11721045168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: gpt Disk identifier: E799A1D5-F9B7-C843-AB62-AADC9B0A2180 Device Start End Sectors Size Type /dev/sdd1 2048 11721027583 11721025536 5.5T Solaris /usr & Apple ZFS /dev/sdd9 11721027584 11721043967 16384 8M Solaris reserved 1
Notice this: the mirror was originally created in FreeNAS (FreeBSD). The sdc has a 2G swap at the beginning of the drive. sdd was created in Ubuntu, and for whatever reason was given an 8M swap at the end of the drive.
Now, fearing that the issue was a bad drive, I offlined sdd and ran badblocks on it. This wipes all info. The good news is that the drive is fine, no bad blocks. This also resets the partitions to nothing.
I have two choices. 1.) Try to manually match the partitions of sdd to the working drive (sdc). Although I thought zfs was supposed to do this automatically by just doing a zpool replace, so maybe that's a waste of time. 2.) I have the data backed up, so I could wipe both drives and start from scratch, create a new mirror, and let it be a native Ubuntu pool.
Maybe this is overthinking it, but I would think I run a bit more risk destroying and restoring. I'm having to destroy good data that's only backed up on a non-mirrored disk, then rsyncing it back to a newly created pool. FYI, I used rsync to create the backup, and it's on the same pc. I had to stripe 3 drives together with no redundancy to fit all the data on it. I'm also afraid that moving the data around like that with no way to scrub it might give me some corruption and I would never know.
Anyone have any advice? Thanks!