2

I am running Arch Linux on a system with full disk encryption and the storage is a Samsung 950 Pro NVMe drive (512 GB). The computer is a couple months old. No bad behavior until now. (I'm only using 21 GB of the 512 space on the disk.)

btrfs-progs v4.5.1

Today I was using my system normally and browsing the web. Firefox stopped responding suddenly and for no apparent reason. Then plasma stopped responding. I could not log out of KDE.

I killed my user session (pkill -u me), then I tired to startx. At that point I noticed my root filesystem was read-only.

As a first step, I rebooted. That didn't help anything.

The root fs will not mount. Here is the error info:

running hook [encrypt]
performing fsck on /dev/mapper/cryptroot
mounting /dev/mapper/cryptroot on real root
BTRFS error (device dm-0) in btrfs_replay_log: 2421: errno=-22 unknown (failed to recover log tree)
BTRFS error (device dm-0): cleaner transaction attach returned -30
BTRFS open_ctree failed
mount: wrong fs type, bad option, bad superblock on /dev/mapper/cryptroot
dropped into emergency shell

I typed this by hand, so minor punctuation is the messages is not exact, etc.

journalctl -p3 doesn't report anything that looks related to this.

EDIT: multiple reboots don't resolve it. The error is still the same.

I booted into a LiveUSB environment and ran this command:

cryptsetup open --type luks /dev/xxx cryptroot
mount -t btrfs -o noatime,nodiratime,ssd,compress=lzo,defaults,space_cache,subvolid=257 /dev/mapper/cryptroot /mnt

The error message is also the same:

[ 2300.967048] BTRFS info (device dm-0): use ssd allocation scheme
[ 2300.967058] BTRFS info (device dm-0): use lzo compression
[ 2300.967066] BTRFS info (device dm-0): disk space caching is enabled
[ 2300.967069] BTRFS: has skinny extents
[ 2300.995393] BTRFS: error (device dm-0) in btrfs_replay_log:2413: errno=-22 unknown (Failed to recover log tree)
[ 2300.997617] BTRFS info (device dm-0): delayed_refs has NO entry
[ 2300.997673] BTRFS error (device dm-0): cleaner transaction attach returned -30
[ 2301.035405] BTRFS: open_ctree failed

EDIT 2:

Based on these two links:

https://btrfs.wiki.kernel.org/index.php/Problem_FAQ
https://btrfs.wiki.kernel.org/index.php/Btrfs-zero-log

I decided to take a chance on running this command:

btrfs rescue zero-log

That worked and I can mount the filesystem.

EDIT 3:

I ran btrfs check --repair. Here is the output:

root@broken / # umount /mnt 
root@broken / # btrfs check --repair /dev/mapper/cryptroot 
enabling repair mode
Checking filesystem on /dev/mapper/cryptroot
checking extents
bad metadata [292414476288, 292414492672) crossing stripe boundary
bad metadata [292414541824, 292414558208) crossing stripe boundary
bad metadata [292414672896, 292414689280) crossing stripe boundary
bad metadata [292414869504, 292414885888) crossing stripe boundary
bad metadata [292415000576, 292415016960) crossing stripe boundary
bad metadata [292415066112, 292415082496) crossing stripe boundary
bad metadata [292415131648, 292415148032) crossing stripe boundary
bad metadata [292415262720, 292415279104) crossing stripe boundary
bad metadata [292415328256, 292415344640) crossing stripe boundary
bad metadata [292415393792, 292415410176) crossing stripe boundary
repaired damaged extent references
Fixed 0 roots.
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
checking quota groups
Ignoring qgroup relation key 258
Ignoring qgroup relation key 263
Ignoring qgroup relation key 71776119061217538
Ignoring qgroup relation key 71776119061217543
Counts for qgroup id: 257 are different
our:            referenced 10412273664 referenced compressed 10412273664
disk:           referenced 10411311104 referenced compressed 10411311104
diff:           referenced 962560 referenced compressed 962560
our:            exclusive 10412273664 exclusive compressed 10412273664
disk:           exclusive 10412273664 exclusive compressed 10412273664
found 21570773057 bytes used err is 0
total csum bytes: 19563456
total tree bytes: 403767296
total fs tree bytes: 349667328
total extent tree bytes: 27328512
btree space waste bytes: 66313360
file data blocks allocated: 39882014720
referenced 28043988992
extent buffer leak: start 20987904 len 16384
extent buffer leak: start 292688068608 len 16384
extent buffer leak: start 60915712 len 16384
extent buffer leak: start 29569581056 len 16384
extent buffer leak: start 29569597440 len 16384
extent buffer leak: start 292412063744 len 16384
extent buffer leak: start 292405870592 len 16384
extent buffer leak: start 292405936128 len 16384
extent buffer leak: start 292413964288 len 16384

Then I checked dmesg and saw this error information:

[ 4925.562422] BTRFS info (device dm-0): use ssd allocation scheme
[ 4925.562432] BTRFS info (device dm-0): use lzo compression
[ 4925.562440] BTRFS info (device dm-0): disk space caching is enabled
[ 4925.562444] BTRFS: has skinny extents
[ 4925.578705] BTRFS error (device dm-0): qgroup generation mismatch, marked as inconsistent
[ 4925.584033] BTRFS: checking UUID tree

My questions:

  1. What caused this?
  2. What should I do next?

Off-target Reference: https://wiki.archlinux.org/index.php/Btrfs#BTRFS:_open_ctree_failed (that doesn't seem to be my problem. I don't have a multi-device btrfs fs and I am not using btrfs in HOOKS in /etc/mkinitcpio.conf)

  • You remember that when you did mkfs.brtfs ... it printed a list of superblock inodes, right? Is there any chance that you saved that list somewhere? – grochmal Aug 6 '16 at 0:41
  • Thanks for replying. Yes, I did save the output from running mkfs.btrfs. There was a UUID, the node size, sector size, fs size and block group sizes. A list of superblock nodes is definitely NOT in my saved notes. Either I missed it or it wasn't there. Suggested next steps? – MountainX Aug 6 '16 at 0:49
  • Well, you have the disk in /dev/mapper/..., which means that the LUKS is alright. This may sound stupid, but have you tried just a couple of plain reboots? You said this is a new machine, encrypted disks suffer when the RAM module is defective in any way (and, on a new machine that's not totally unlikely). What I'd try is to run a RAM check (every liveCD provides one) for a long time (several hours) to see if it finds any issue in the RAM modules. If it does find issues (even little ones) you will not e able to run a fully encrypted system on this RAM. – grochmal Aug 6 '16 at 0:57
  • I also ran cryptsetup status /dev/mapper/cryptroot and it looked normal, so I agree that LUKs is probably OK. I'll try several reboots and see what happens. I've been using the machine for several months and it has not exhibited any strange behavior. The RAM is a matched set of 64 GB low latency (expensive) sticks from a good brand. – MountainX Aug 6 '16 at 1:01
  • I updated the question with more info. – MountainX Aug 6 '16 at 1:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.