4

I am struggling to know how to fix an error I have on a btrfs partition. For a reason maybe related to having to switch off my computer a few times due to complete lock up (graphic problems I think), I suddenly found that chrome (stable) would no longer work. I was able to install chrome unstable, and after a little digging discovered that my ~/.config/google-chrome directory was rubbish (ls ~/.config showed ?s all over that line.). I tried deleting the directory but all I kept getting was i/o errors.

btrfs scrub start -Bd /home/alan fails to complete

ERROR: scrubbing /home/alan failed for device id 1: ret=-1, errno=5 (Input/output error)
scrub device /dev/nvme0n1p4 (id 1) canceled
Scrub started:    Wed May 27 09:11:37 2020
Status:           aborted
Duration:         0:00:51
Total to scrub:   199.02GiB
Rate:             3.24GiB/s
Error summary:    csum=1
  Corrected:      0
  Uncorrectable:  1
  Unverified:     0

However if I run sudo smartctl -H /dev/nvme0n1 (this device holding this and other partitions) it reports it as completely healthy.

If I look at the output from journalctl for BTRFS errors a particular combination shows up quite frequently

BTRFS warning (device nvme0n1p4): checksum error at logical 174610628608 on dev /dev/nvme0n1p4, physical 174610628608: metadata leaf (level 0) in tree 263
BTRFS warning (device nvme0n1p4): checksum error at logical 174610628608 on dev /dev/nvme0n1p4, physical 174610628608: metadata leaf (level 0) in tree 174307803136
BTRFS warning (device nvme0n1p4): checksum error at logical 174610628608 on dev /dev/nvme0n1p4, physical 174610628608: metadata leaf (level 0) in tree 1522483200
BTRFS warning (device nvme0n1p4): checksum error at logical 174610628608 on dev /dev/nvme0n1p4, physical 174610628608: metadata leaf (level 0) in tree 1423638528
BTRFS warning (device nvme0n1p4): checksum error at logical 174610628608 on dev /dev/nvme0n1p4, physical 174610628608: metadata leaf (level 0) in tree 1333559296
BTRFS warning (device nvme0n1p4): checksum error at logical 174610628608 on dev /dev/nvme0n1p4, physical 174610628608: metadata leaf (level 0) in tree 1252245504
BTRFS warning (device nvme0n1p4): checksum error at logical 174610628608 on dev /dev/nvme0n1p4, physical 174610628608: metadata leaf (level 0) in tree 1252245504
BTRFS error (device nvme0n1p4): bdev /dev/nvme0n1p4 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
BTRFS error (device nvme0n1p4): unable to fixup (regular) error at logical 174610628608 on dev /dev/nvme0n1p4

It tried reading it with sudo hdparm --read-sector 174610628608 /dev/nvme0n1p4

but that failed

/dev/nvme0n1p4:
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
Device /dev/nvme0n1p4 has non-zero LBA starting offset of 161134592.
Please use an absolute LBA with the /dev/ entry for the raw device, rather than a partition or raid name.
/dev/nvme0n1p4 is probably a partition of /dev/nvme0n1p (?)
The absolute LBA of sector 174610628608 from /dev/nvme0n1p4 should be 174771763200
Aborting.

I also tried with the other number it gave on the whole device sudo hdparm --read-sector 174771763200 /dev/nvme0n1 but that just failed more simply

/dev/nvme0n1:
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
reading sector 174771763200: FAILED: Inappropriate ioctl for device

Where do I go from here. Its going to be a major distruption to completely reformat the entire partition (even worse for the entire device) as this partition also has a separate btrfs subvolume that is the rootfs for my debian system. I will of course ensure I have backups on another device first, but is there any other way?

ADDING ANSWERS TO COMMENTS ...

Response to dmesg -T | tail -n 20:-

[Wed May 27 10:53:11 2020] docker0: port 2(veth79249a8) entered forwarding state
[Wed May 27 10:53:11 2020] docker0: port 2(veth79249a8) entered disabled state
[Wed May 27 10:53:11 2020] eth0: renamed from veth5edb139
[Wed May 27 10:53:11 2020] IPv6: ADDRCONF(NETDEV_CHANGE): veth79249a8: link becomes ready
[Wed May 27 10:53:11 2020] docker0: port 2(veth79249a8) entered blocking state
[Wed May 27 10:53:11 2020] docker0: port 2(veth79249a8) entered forwarding state
[Wed May 27 10:53:11 2020] eth0: renamed from veth3e5602d
[Wed May 27 10:53:11 2020] IPv6: ADDRCONF(NETDEV_CHANGE): veth91ddda7: link becomes ready
[Wed May 27 10:53:11 2020] docker0: port 1(veth91ddda7) entered blocking state
[Wed May 27 10:53:11 2020] docker0: port 1(veth91ddda7) entered forwarding state
[Wed May 27 10:53:16 2020] rfkill: input handler enabled
[Wed May 27 10:53:16 2020] systemd-journald[405]: File /var/log/journal/e76b13bb9e67491ca8c1ce766232fa75/user-1000.journal corrupted or uncleanly shut down, renaming and replacing.
[Wed May 27 10:53:18 2020] rfkill: input handler disabled
[Wed May 27 15:07:42 2020] BTRFS error (device nvme0n1p4): bad tree block start, want 174610628608 have 0
[Wed May 27 15:49:36 2020] ThreadPoolForeg[57126]: segfault at 312d1a400010 ip 000055daf8b8c542 sp 00007fc0c2cd6f90 error 4 in chrome[55daf6e12000+7a42000]
[Wed May 27 15:49:36 2020] Code: 8b 7c 0a 10 41 f6 c7 01 0f 84 81 00 00 00 4c 8b 54 0a 08 4d 89 fe 49 81 e6 00 00 fc ff 44 89 f9 c0 e9 02 bb 01 00 00 00 d3 e3 <4d> 8b 46 10 44 89 f9 c1 e9 07 81 e1 ff 07 00 00 41 8b 04 88 89 c7
[Wed May 27 15:58:01 2020] chrome[56261]: segfault at 25a517607ac0 ip 000055dafd7fb6ce sp 00007ffcd366ec10 error 4 in chrome[55daf6e12000+7a42000]
[Wed May 27 15:58:01 2020] Code: 0f 0b cc cc cc cc cc cc cc cc cc cc 55 48 89 e5 41 57 41 56 53 50 49 89 fe 48 8b 5f 28 48 c7 47 28 00 00 00 00 48 85 db 74 10 <48> 8b 3b 48 85 ff 75 5d 48 89 df e8 d2 45 92 f9 49 8b 5e 20 48 85
[Wed May 27 17:23:01 2020] Chrome_IOThread[6934]: segfault at 1841cd6f82c8 ip 000055587adb8a5e sp 00007fece6cc7990 error 6 in chrome[55587aa9f000+7a42000]
[Wed May 27 17:23:01 2020] Code: 00 00 4d 39 ee 0f 84 83 03 00 00 4c 89 f8 4c 31 f0 48 f7 d0 49 89 45 00 4d 39 ee 0f 84 d2 03 00 00 4c 89 f8 4c 31 e8 48 f7 d0 <49> 89 46 08 4d 85 ed 0f 85 05 ff ff ff c7 45 c0 03 00 00 00 4c 89
6
  • The BTRFS errors you show basically indicate that there's damage to one or more metadata chunks. Presumably you don't have metadata redundancy (which is disabled by default for single-SSD filesystems) which is why BTRFS could not repair the corruption. Is the kernel reporting I/O errors from the device as well, or are the errors only from BTRFS? May 27, 2020 at 11:36
  • BTRFS records an entry in kern.log, but no other indication from the kernel itself. I don't need to recover the files in that directory as I have a backup from earlier. Is there a way tell BTRFS to remove the directory entry.
    – akc42
    May 27, 2020 at 14:10
  • Anything in dmesg -T | tail -n 20? Would be interesting to know the real problem scrub runs into: strace -tt -f btrfs scrub start -Bd /home/alan 2>&1 | tail -n 1000 >scrub.strace Look for errors near the end. Have you tried reading the sector with dd or running btrfs check? I guess the sector with the checksum error does not have to be the one with the I/O error. May 27, 2020 at 15:27
  • I've edit my question to include the results of dmesg -T |tail -n 20, I ran strace but the output was just repeated sections of the same thing, except at the end of one the number of uncorrectable errors became 1 and the brtrs scrub shutdown started. No indication of what the error was. I' have to do the btrfs check tomorrow
    – akc42
    May 27, 2020 at 21:29
  • 1
    @HaukeLaging Sorry for missing that off. It isn't good news I did try a btrfs check --repair but that just made it worse. I've had to clear the disk and start again from backups. I did get the installer to write zeros over the whole partition just to check it could write everywhere and not get any low level io errors. Am at least got enough restarted to login in here and respond - still someway to go to restore everything though. Thanks for the help and it has made me wonder about relying on btrfs so heavily. I am wondering if I can (and therefore should) duplicate the metadata.
    – akc42
    May 28, 2020 at 10:15

0

You must log in to answer this question.

Browse other questions tagged .