3

I have Btrfs setup on 3 disks with metadata and data in RAID1. But now I have a checksum error which it cannot recover.

The checksum is the same on both copies and only differs from the expected checksum by one flipped bit. Therefore I suspect there was a bitflip on the checksum before it was written to the disks (the computer does not have ECC RAM). I have a copy of the actual file on another computer from before it was written to this filesystem but as shown below I cannot read out the data due to I/O error from the filesystem so I cannot compare them.

How should I go on to fix this error?

Some details:

$ uname -a
Linux stan 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ btrfs --version
btrfs-progs v4.15.1
$ sudo btrfs fi usage /media/btrfs/
Overall:
    Device size:           7.28TiB
    Device allocated:          3.91TiB
    Device unallocated:        3.36TiB
    Device missing:          0.00B
    Used:              3.83TiB
    Free (estimated):          1.72TiB  (min: 1.72TiB)
    Data ratio:               2.00
    Metadata ratio:           2.00
    Global reserve:      512.00MiB  (used: 0.00B)

Data,RAID1: Size:1.95TiB, Used:1.91TiB
   /dev/sdb    1.95TiB
   /dev/sdc  998.00GiB
   /dev/sdd 1001.00GiB

Metadata,RAID1: Size:4.00GiB, Used:2.63GiB
   /dev/sdb    4.00GiB
   /dev/sdc    3.00GiB
   /dev/sdd    1.00GiB

System,RAID1: Size:64.00MiB, Used:304.00KiB
   /dev/sdb   64.00MiB
   /dev/sdc   64.00MiB

Unallocated:
   /dev/sdb    1.68TiB
   /dev/sdc  861.95GiB
   /dev/sdd  861.02GiB

Scrub:

$ sudo btrfs scrub status /media/btrfs/

scrub status for xxxxxx
    scrub started at Mon Aug 24 11:23:27 2020 and finished after 03:41:54
    total bytes scrubbed: 3.81TiB with 2 errors
    error details: csum=2
    corrected errors: 0, uncorrectable errors: 2, unverified errors: 0

Dmesg error after scrub.

$ dmesg
...
196755.786038] BTRFS warning (device sdb): checksum error at logical 3099310968832 on dev /dev/sdb, physical 1300730499072, root 5223, inod
e 6521311, offset 7614464, length 4096, links 1 (path: users/joachim/Bilder/Canon/270CANON/IMG_7003.CR2)
[196755.786168] BTRFS warning (device sdb): checksum error at logical 3099310968832 on dev /dev/sdb, physical 1300730499072, root 5303, inod
e 6521311, offset 7614464, length 4096, links 1 (path: users/joachim/Bilder/Canon/270CANON/IMG_7003.CR2)
[196755.786245] BTRFS warning (device sdb): checksum error at logical 3099310968832 on dev /dev/sdb, physical 1300730499072, root 5302, inod
e 6521311, offset 7614464, length 4096, links 1 (path: users/joachim/Bilder/Canon/270CANON/IMG_7003.CR2)
...
[196755.788274] BTRFS error (device sdb): bdev /dev/sdb errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[196755.814044] BTRFS error (device sdb): unable to fixup (regular) error at logical 3099310968832 on dev /dev/sdb

Inspect-internal on block:

$ sudo btrfs inspect-internal logical-resolve -v 3099310968832 /media/btrfs/
ioctl ret=0, total_size=4096, bytes_left=3456, bytes_missing=0, cnt=78, missed=0
ioctl ret=0, bytes_left=4023, bytes_missing=0, cnt=1, missed=0
/media/btrfs//snapshots/stansafe.20200601T032501+0200/users/joachim/Bilder/Canon/270CANON/IMG_7003.CR2
ioctl ret=0, bytes_left=4023, bytes_missing=0, cnt=1, missed=0
/media/btrfs//snapshots/stansafe.20200910T032501+0200/users/joachim/Bilder/Canon/270CANON/IMG_7003.CR2
ioctl ret=0, bytes_left=4023, bytes_missing=0, cnt=1, missed=0
/media/btrfs//snapshots/stansafe.20200909T032502+0200/users/joachim/Bilder/Canon/270CANON/IMG_7003.CR2
...

Trying to verify the file:

$ sha256sum /media/btrfs//stansafe/users/joachim/Bilder/Canon/270CANON/IMG_7003.CR2
sha256sum: /media/btrfs//stansafe/users/joachim/Bilder/Canon/270CANON/IMG_7003.CR2: Input/output error

$ dmesg
...
[1642985.509498] BTRFS warning (device sdb): csum failed root 259 ino 6521311 off 7614464 csum 0x151ad4ce expected csum 0x150ad4ce mirror 1
[1642985.509942] BTRFS warning (device sdb): csum failed root 259 ino 6521311 off 7614464 csum 0x151ad4ce expected csum 0x150ad4ce mirror 2

2 Answers 2

0

Linux Kernel 5.11 introduced the option to mount and ignore file checksums. This will allow copying out data with bad csums.

mount -o rescue=ignoredatacsums /dev/sdX /mnt

This could let you completely recover a file if it is minimally damaged and you have some kind of parity, like par2

Source: https://btrfs.readthedocs.io/en/latest/btrfs-man5.html

1
  • Thank you for the reply it might help me in the future. My solution ended up being writing a utility myself: github.com/jlublin/btrfs-plumber It was only errors in the checksums so just changing those values fixed the problems.
    – jlublin
    May 19, 2022 at 19:30
-1

if you want to store 3 copies of all files on 3 hard disk or on 3 partitions

use raid1c3

that way if you loose even 2 hard disks , your data in 1 hard disk will be still there .

raid1 is for storing only 2 copies no matter how many hard disks you use

1
  • This does not appear to address the issue.
    – Kusalananda
    Nov 28, 2023 at 10:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .