btrfs has invalid csums, scrubbing aborts with I/O errors but SSD seems fine

Question

I've ben seeing an issue where updates on my openSUSE Tumbleweed machine fail after some packages have been installed, claiming that /var was a read-only file system.

I've reverted to earlier snapshots, tested that /var is not read-only, re-ran updates, and after some error message it's back to read-only.

This question led me to check the boot messages, and wouldn't you know there's an issue with BTRFS:

[  231.762975] BTRFS info (device sda2): scrub: started on devid 1
[  287.021834] BTRFS error (device sda2): parent transid verify failed on 31572885504 wanted 278272 found 278280
[  287.060064] BTRFS info (device sda2): scrub: not finished on devid 1 with status: -5
[  643.134491] BTRFS info (device sda2): qgroup scan completed (inconsistency flag cleared)
[  971.347644] BTRFS info (device sda2): scrub: started on devid 1
[ 1026.335159] BTRFS error (device sda2): parent transid verify failed on 31572885504 wanted 278272 found 278280
[ 1026.374518] BTRFS info (device sda2): scrub: not finished on devid 1 with status: -5

The last 3 lines repeat once more. Switching to an ealier snapshot seems to affect nothing, so it's probably not some recent change to the file system contents which may have been interrupted half-way. It's either been in existence for some time, or it's something different.

I tried scrubbing, but that aborts a minute (and about 14 GiB) into the process, with an I/O error:

> sudo btrfs scrub start -B /dev/sda2
ERROR: scrubbing /dev/sda2 failed for device id 1: ret=-1, errno=5 (Input/output error)
scrub canceled for 8b283f24-277b-4cf8-8d87-6107bca1ef57
Scrub started:    Wed Jul 15 14:20:22 2020
Status:           aborted
Duration:         0:00:55
Total to scrub:   60.00GiB
Rate:             183.09MiB/s
Error summary:    no errors found

So, no errors found, but aborted due to I/O error? Looks like there was an error after all.

I did test the S.M.A.R.T. status of the drive, and it seems to be completely fine, as far as I can tell. The drive has about 2700h of service, so I wouldn't expect a lot of wear on it, either.

I've searched some more and found this which recommends essentially replacing the contents of the disk from a backup. Since this is my main system partition, I'm not at all looking forward to installing the whole thing over. I do have a recent partclone backup but that also has the errors (which have probably been around for a while). Also: My system works fine unless I try to update something, so maybe it can be salvaged in some way?

Just checking for csum errors:

> sudo btrfs check --check-data-csum /dev/sda2
Opening filesystem to check...
WARNING: filesystem mounted, continuing because of --force
Checking filesystem on /dev/sda2
UUID: 8b283f24-277b-4cf8-8d87-6107bca1ef57
[1/7] checking root items
[2/7] checking extents
parent transid verify failed on 31572885504 wanted 278272 found 278280
parent transid verify failed on 31572885504 wanted 278272 found 278280
Ignoring transid failure
[3/7] checking free space cache
[4/7] checking fs roots
parent transid verify failed on 31572885504 wanted 278272 found 278280
Ignoring transid failure
parent transid verify failed on 31572885504 wanted 278272 found 278280
Ignoring transid failure
parent transid verify failed on 31572885504 wanted 278272 found 278280
Ignoring transid failure
root 259 inode 4735696 errors 800, odd csum item
root 259 inode 4746779 errors 800, odd csum item
root 259 inode 4747724 errors 800, odd csum item
parent transid verify failed on 31572885504 wanted 278272 found 278280
Ignoring transid failure
[... lots of repetitions of the previous two lines ...]
Ignoring transid failure
ERROR: errors found in fs roots
found 49867616256 bytes used, error(s) found
total csum bytes: 38229736
total tree bytes: 1010974720
total fs tree bytes: 895434752
total extent tree bytes: 57819136
btree space waste bytes: 215524051
file data blocks allocated: 869778038784
 referenced 68509286400

Soo ... does this mean it's just one single block that's affected by the checksum issue? Would that mean it's only a single file? Or does the "errors found in fs roots" line indicate more things wrong with the file system?

I've seen this advice to use zero-log but that command does not seem to exist neither on my system nor on the Manjaro Live system I used to check the disk while unmounted, su I suppose that's no longer needed/supported? Regardless, the btrfs wiki says that zero-log is useless as long as the file system can be mounted, and mine can be mounted. However, it also says not to use btrfs-check for repairs unless all else fails.

To me it seems as if all else has indeed failed, however I am not sure if I overlooked some other way to deal with the issue, and maybe even work out what went wrong in the first place.

So, is it worth trying to fix this via btrfs check --init-csum-tree (or btrfs check --repair?) or is there a smarter way to fix this without re-installing the system? Maybe pin-point which file(s) are affected and checking whether they could be fixed or re-generated?

Answer 1

I did run whatever other analysis methods I could find to try and work out which files were affected by the transid failures, but could not really get very far. So I made a backup of anything still readable using btrfs restore, and ran btrfs check --repair and btrfs check --init-csum-tree, from a live-booted system. That got rid of the error reports, but left very little space indicated as free. So I followed up with brfs balance for good measure (had to run several times, first time limited to almost-empty blocks (usage=10) because free space was rare. Several scrubs and balances later, the drive seemed to work normally again, but some files were broken/lost. I uninstalled/reinstalled some of the affected packages which didn't work anymore, ran a full system update, and it's all good again.

To reduce the likelyhood of such errors happening again (and staying unnoticed), I've now set up a service in openSUSE to regularly scrub and balance the disk. Running fine since then. I really wish that this kind of hygiene measures were part of BTRFS: Scrub every X writes, balance after Y% of blocks have been reallocated ... and raise a flag if something goes wrong. Or at least the distros which offer BTRFS by default should have something along those lines pre-configured because anyone using BTRFS needs to scrub and balance anyway.