3

Environment

$ btrfs --version
btrfs-progs v4.2.1

$ uname -a
Linux [REDACTED] 4.2.1-040201-generic #201509211431 SMP Mon Sep 21 18:34:44 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Problem

Today the controller of one of my external drives died. The disk is alright though. I cracked open the case, got the drive out and connected it via a SATA-USB interface.

Now, mounting the filesystem is not possible. Here's the message:

$ btrfs fi show
warning devid 3 not found already
Label: none  uuid: bd6090df-5179-490e-a5f8-8fbad433657f
        Total devices 3 FS bytes used 3.02TiB
        devid    1 size 596.17GiB used 532.03GiB path /dev/sdd
        devid    2 size 931.51GiB used 867.03GiB path /dev/sde
        *** Some devices missing

Yes, I did bundle up three drives with very different sizes with the --single option on creating the file system. This absolutely works.

However, now:

$ mount /dev/sdd /mnt/btrfs/
mount: wrong fs type, bad option, bad superblock on /dev/sdd,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

Even following the official recommendation to mount degraded does not work. Here's a dmesg excerpt:

[ 1849.938705] BTRFS info (device sdd): allowing degraded mounts
[ 1849.938716] BTRFS info (device sdd): disk space caching is enabled
[ 1849.938720] BTRFS: has skinny extents
[ 1850.345011] BTRFS: bdev (null) errs: wr 2033, rd 1074, flush 0, corrupt 0, gen 249
[ 1854.953599] BTRFS: too many missing devices, writeable mount is not allowed
[ 1854.992299] BTRFS: open_ctree failed

I'm left with what appears to be a completely broken filesystem, although all disks are in working order. It's kind of a catch 22:

  • The drive with the broken controller is recognized but will not join the btrfs filesystem.
  • I can mount read-only but that doesn't allow me to add or remove disks.

Is there any way to add the "missing" disk to the filesystem manually, ideally without removing it's data content?

I'm happy to hear any ideas.

Update

/dev/sdb:

$ dd iflag=skip_bytes skip=64K bs=320 count=1 if=/dev/sdb | xxd
0000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
1+0 records in
1+0 records out
0000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
320 bytes (320 B) copied0000030: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000040: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000050: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000060: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000070: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000080: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000090: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000a0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000b0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
, 0.000956237 s, 335 kB/s
00000c0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000d0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000e0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000f0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000100: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000110: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000120: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000130: 0000 0000 0000 0000 0000 0000 0000 0000  ................

$ head -c 512 /dev/sdb | xxd
0000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000030: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000040: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000050: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000060: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000070: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000080: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000090: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000a0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000b0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000c0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000d0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000e0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000f0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000100: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000110: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000120: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000130: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000140: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000150: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000160: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000170: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000180: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000190: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00001a0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00001b0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00001c0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00001d0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00001e0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00001f0: 0000 0000 0000 0000 0000 0000 0000 0000  ................

/dev/sde:

$ dd iflag=skip_bytes skip=64K bs=320 count=1 if=/dev/sde | xxd
1+0 records in
1+0 records out
320 bytes (320 B) copied, 0.000681015 s, 470 kB/s
0000000: 6c7f 1e0d 0000 0000 0000 0000 0000 0000  l...............
0000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000020: bd60 90df 5179 490e a5f8 8fba d433 657f  .`..QyI......3e.
0000030: 0000 0100 0000 0000 0100 0000 0000 0000  ................
0000040: 5f42 4852 6653 5f4d 0778 0000 0000 0000  _BHRfS_M.x......
0000050: 00c0 7288 e302 0000 0080 c145 4404 0000  ..r........ED...
0000060: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000070: 00c0 1083 4f03 0000 0010 d704 0403 0000  ....O...........
0000080: 0600 0000 0000 0000 0300 0000 0000 0000  ................
0000090: 0010 0000 0040 0000 0040 0000 0010 0000  .....@...@......
00000a0: 8100 0000 c076 0000 0000 0000 0000 0000  .....v..........
00000b0: 0000 0000 0000 0000 0000 0000 6101 0000  ............a...
00000c0: 0000 0000 0000 0101 0002 0000 0000 0000  ................
00000d0: 0000 60db e0e8 0000 0000 0000 c2d8 0000  ..`.............
00000e0: 0000 1000 0000 1000 0000 1000 0000 0000  ................
00000f0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000100: 0000 0000 0000 0000 0000 009c 07fd 3a49  ..............:I
0000110: c047 079e f03b 23da 5f58 f6bd 6090 df51  .G...;#._X..`..Q
0000120: 7949 0ea5 f88f bad4 3365 7f00 0000 0000  yI......3e......
0000130: 0000 0000 0000 0000 0000 0000 0000 0000  ................

$ head -c 512 /dev/sde | xxd
0000000: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000030: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000040: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000050: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000060: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000070: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000080: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000090: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000a0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000b0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000c0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000d0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000e0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000f0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000100: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000110: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000120: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000130: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000140: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000150: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000160: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000170: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000180: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000190: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00001a0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00001b0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00001c0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00001d0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00001e0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00001f0: 0000 0000 0000 0000 0000 0000 0000 0000  ................

Update 2:

fdisk -l /dev/sd? output:

Disk /dev/sda: 256.1 GB, 256060514304 bytes
255 heads, 63 sectors/track, 31130 cylinders, total 500118192 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1     1050623      525311+  ee  GPT
/dev/sda2   *     1050624     1550335      249856   83  Linux
/dev/sda3         1550336   500117503   249283584   83  Linux

Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Note: sector size is 4096 (not 512)

Disk /dev/sdc: 4000.8 GB, 4000752599040 bytes
164 heads, 36 sectors/track, 165438 cylinders, total 976746240 sectors
Units = sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x05e82c7f

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1             256   976746239  3906983936   83  Linux

Disk /dev/sdd: 640.1 GB, 640135028736 bytes
255 heads, 63 sectors/track, 77825 cylinders, total 1250263728 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/sde: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Update 3:

$ strings -a -8 -t d /dev/sde | grep -m 1 '_BHRfS_M'
  65600 _BHRfS_M
$ strings -a -8 -t d /dev/sdd | grep -m 1 '_BHRfS_M'
  65600 _BHRfS_M
$ strings -a -8 -t d /dev/sdc | grep -m 1 '_BHRfS_M'
1114176 _BHRfS_M

Update 4:

$ dd iflag=skip_bytes skip=64K bs=320 count=1 if=/dev/sdb | xxd
1+0 records in
1+0 records out
0000000: dd85 1110 1c99 5a1d 7ee0 53f9 0d39 071a  ......Z.~.S..9..
320 bytes (320 B) copied0000010: dd85 1110 1c99 5a1d 7ee0 53f9 0d39 071a  ......Z.~.S..9..
0000020: dd85 1110 1c99 5a1d 7ee0 53f9 0d39 071a  ......Z.~.S..9..
0000030: dd85 1110 1c99 5a1d 7ee0 53f9 0d39 071a  ......Z.~.S..9..
0000040: dd85 1110 1c99 5a1d 7ee0 53f9 0d39 071a  ......Z.~.S..9..
, 0.00123336 s, 259 kB/s
0000050: dd85 1110 1c99 5a1d 7ee0 53f9 0d39 071a  ......Z.~.S..9..
0000060: dd85 1110 1c99 5a1d 7ee0 53f9 0d39 071a  ......Z.~.S..9..
0000070: dd85 1110 1c99 5a1d 7ee0 53f9 0d39 071a  ......Z.~.S..9..
0000080: dd85 1110 1c99 5a1d 7ee0 53f9 0d39 071a  ......Z.~.S..9..
0000090: dd85 1110 1c99 5a1d 7ee0 53f9 0d39 071a  ......Z.~.S..9..
00000a0: dd85 1110 1c99 5a1d 7ee0 53f9 0d39 071a  ......Z.~.S..9..
00000b0: dd85 1110 1c99 5a1d 7ee0 53f9 0d39 071a  ......Z.~.S..9..
00000c0: dd85 1110 1c99 5a1d 7ee0 53f9 0d39 071a  ......Z.~.S..9..
00000d0: dd85 1110 1c99 5a1d 7ee0 53f9 0d39 071a  ......Z.~.S..9..
00000e0: dd85 1110 1c99 5a1d 7ee0 53f9 0d39 071a  ......Z.~.S..9..
00000f0: dd85 1110 1c99 5a1d 7ee0 53f9 0d39 071a  ......Z.~.S..9..
0000100: dd85 1110 1c99 5a1d 7ee0 53f9 0d39 071a  ......Z.~.S..9..
0000110: dd85 1110 1c99 5a1d 7ee0 53f9 0d39 071a  ......Z.~.S..9..
0000120: dd85 1110 1c99 5a1d 7ee0 53f9 0d39 071a  ......Z.~.S..9..
0000130: dd85 1110 1c99 5a1d 7ee0 53f9 0d39 071a  ......Z.~.S..9.

Update 5:

For what it's worth:

$ btrfs-show-super -a /dev/sdb
superblock: bytenr=65536, device=/dev/sdb
---------------------------------------------------------
ERROR: bad magic on superblock on /dev/sdb at 65536

superblock: bytenr=67108864, device=/dev/sdb
---------------------------------------------------------
ERROR: bad magic on superblock on /dev/sdb at 67108864

superblock: bytenr=274877906944, device=/dev/sdb
---------------------------------------------------------
ERROR: bad magic on superblock on /dev/sdb at 274877906944
  • what kind of controller was it? some raid controllers take (and hide) the first few sectors of the disk to store information about the disk's setup....if you later plug the disk into a standard controller, the disk will appear to have no partition table. – cas Sep 22 '15 at 1:09
  • What is the output of file -s /dev/<device>? – Vincent Yu Sep 22 '15 at 2:27
  • @cas: A simple Western Digital SATA controller, built in a WD Studio external enclosure. It could of course be that those controllers do things differently than a standard controller. – herrbischoff Sep 22 '15 at 12:15
  • @VincentYu: /dev/sdb: data, while the others output /dev/sde: BTRFS Filesystem sectorsize 4096, nodesize 16384, leafsize 16384, UUID=bd6090df-5179-490e-a5f8-8fbad433657f, 3315795955712/3640036212736 bytes used, 3 devices – herrbischoff Sep 22 '15 at 12:16
  • 1
    I'm assuming that sdb is the drive in question? Looks like the first btrfs superblock at 64K cannot be recognized; to confirm, can you post the output of dd iflag=skip_bytes skip=64K bs=320 count=1 if=/dev/sdb | xxd? Also check if the mirror superblocks exist by rerunning that command with 64M and 256G instead of 64K and looking for the magic string _BHRfS_M in the output (basic structure of btrfs superblocks). Finally, please post the header of the disk if there is anything there: head -c 512 /dev/sdb | xxd – Vincent Yu Sep 22 '15 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.