7

I noticed that my HP N54L was laboring and discovered that dmesg was reporting this:

[   81.945530] btrfs read error corrected: ino 1 off 16685977600 (dev /dev/sdb sector 2636776)
[   82.010023] btrfs read error corrected: ino 1 off 16637734912 (dev /dev/sdb sector 2589656)

[   85.927604] verify_parent_transid: 43 callbacks suppressed
[   85.927615] parent transid verify failed on 16956989440 wanted 13182 found 12799
[   85.974600] parent transid verify failed on 16585043968 wanted 13145 found 12357

[   89.903548] repair_io_failure: 26 callbacks suppressed
[   89.903560] btrfs read error corrected: ino 1 off 16875483136 (dev /dev/sdb sector 2821816)
[  115.951579] parent transid verify failed on 16963846144 wanted 13184 found 12802
[  115.976830] btrfs read error corrected: ino 1 off 16963846144 (dev /dev/sdb sector 2908128)
[  115.988907] parent transid verify failed on 16978874368 wanted 13187 found 12815

[  543.848294] btrfs: device fsid e8f8fc09-3aae-4fce-85ca-fcf7665b9f02 devid 2 transid 13199 /dev/sdb
[ 1120.854825] verify_parent_transid: 5 callbacks suppressed
[ 1120.854838] parent transid verify failed on 16956600320 wanted 13184 found 12799

[ 1120.891229] repair_io_failure: 6 callbacks suppressed
[ 1120.891243] btrfs read error corrected: ino 1 off 16956600320 (dev /dev/sdb sector 2901016)
[ 1124.851937] parent transid verify failed on 16977842176 wanted 13187 found 12814
[ 1124.885429] btrfs read error corrected: ino 1 off 16977842176 (dev /dev/sdb sector 2921768)

This is my BTRFS setup. RAID10 across 4x3TB HDD:

$ sudo btrfs filesystem df /mnt/btrfs
Data, RAID10: total=136.00GiB, used=134.70GiB
System, RAID10: total=64.00MiB, used=20.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, RAID10: total=1.00GiB, used=363.21MiB

$ sudo btrfs filesystem show /mnt/btrfs
Label: none  uuid: <UUID>
    Total devices 4 FS bytes used 135.05GiB
    devid    1 size 2.73TiB used 68.54GiB path /dev/sda
    devid    2 size 2.73TiB used 68.53GiB path /dev/sdb
    devid    3 size 2.73TiB used 68.53GiB path /dev/sdc
    devid    4 size 2.73TiB used 68.53GiB path /dev/sdd

And I noticed that the device stats from BTRFS were ...odd...:

$ sudo btrfs device stats /mnt/btrfs
[/dev/sda].write_io_errs   0
[/dev/sda].read_io_errs    0
[/dev/sda].flush_io_errs   0
[/dev/sda].corruption_errs 0
[/dev/sda].generation_errs 0
[/dev/sdb].write_io_errs   207275
[/dev/sdb].read_io_errs    127287
[/dev/sdb].flush_io_errs   0
[/dev/sdb].corruption_errs 0
[/dev/sdb].generation_errs 0
[/dev/sdc].write_io_errs   0
[/dev/sdc].read_io_errs    0
[/dev/sdc].flush_io_errs   0
[/dev/sdc].corruption_errs 0
[/dev/sdc].generation_errs 0
[/dev/sdd].write_io_errs   0
[/dev/sdd].read_io_errs    0
[/dev/sdd].flush_io_errs   0
[/dev/sdd].corruption_errs 0
[/dev/sdd].generation_errs 0

I've ordered myself a spare 3TB HDD just in case, but can I safely assume that /dev/sdb is dead? I just found it a little odd that BTRFS was reporting [/dev/sdb].corruption_errs 0.

Is there a universally accepted way of proving that a HDD is dead in a BTRFS RAID array?

1
  • Thanks for the badblocks suggestion. I've begun a read-only test on /dev/sdb. Reason for read-only is because it's still being used by BTRFS and I'm not sure that running badblocks with -n would stop me if it doesn't support BTRFS in general.
    – Socceroos
    Commented Jun 11, 2015 at 16:03

1 Answer 1

5

I have seen similar performance degradations on my server here at home (running RAID-6 with Btrfs on top). It has proven one of the drives on three occasions.

The first thing I do is run smartctl for each drive. Then for the failing drive I notice the number of Raw errors:

smartctl -x /dev/sdf | fgrep Raw

to keep track of those. I have one drive that once showed a few errors but has been stable over the last 9 months after resetting the cabling. Not sure why, but I do consider that one "not yet dead".

If error counts grow again, I remove the drive and bring in replacement (I can live with the risk of one of the two extra drives in my RAID-6 being off line for half a day).

7
  • Not too much difference. /dev/sda: 115282712 /dev/sdb: 146960040 /dev/sdc: 134936032 /dev/sdd: 159359440
    – Socceroos
    Commented Jun 11, 2015 at 15:58
  • Is that the RAW_VALUE last entry of the line) of the Raw_Read_Error_Rate you are getting? They are 0 for all my drives except for the not dead yet one (which is at 46095). Most importantly IMO is that that those numbers should not grow.
    – Anthon
    Commented Jun 11, 2015 at 16:04
  • Yes these numbers are Raw_Read_Error_Rate->RAW_VALUE.
    – Socceroos
    Commented Jun 11, 2015 at 16:07
  • Those seem high to me for any system. But make sure you get a second opinion on that before ordering new hardware ;-)
    – Anthon
    Commented Jun 11, 2015 at 16:11
  • 1
    I've done some reading on smartctl reports and now know a bit more. It seems that the most important values on the Raw_Read_Error_Rate row are VALUE, WORST and THRESH. VALUE is where the drive is currently at - 200 being pretty-much-perfect and 1 being abject-failure. WORST tells you how bad the drive has got in the past - so it will never be higher than VALUE, uses the same scale. THRESH is the lowest number that the manufacturer lets the drive get to before they consider it 'failed'. So, 200 being perfect and 1 being super-failed, /dev/sdb is sitting at 110. THRESH = 6
    – Socceroos
    Commented Jun 11, 2015 at 16:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .