Btrfs RAID1: How to replace a disk drive that is physically no more there?

Question

I have a btrfs RAID1 system with the following state:

# btrfs filesystem show
Label: none  uuid: 975bdbb3-9a9c-4a72-ad67-6cda545fda5e
        Total devices 2 FS bytes used 1.65TiB
        devid    1 size 1.82TiB used 1.77TiB path /dev/sde1
        *** Some devices missing

The missing device is a disk drive that failed completely and which the OS could not recognize anymore. I removed the faulty disk and sent it for recycling.

Now I have a new disk installed under /dev/sdd. Searching the web, I fail to find instructions for such a scenario (bad choice of search terms?). There are many examples how to save a RAID system when the faulty disk still remain somewhat accessible by the OS. btrfs replace command requires a source disk.

I tried the following:

# btrfs replace start 2 /dev/sdd /mnt/brtfs-raid1-b
# btrfs replace status /mnt/brtfs-raid1-b
Never started

No error message, but status indicate it never started. I cannot figure out what the problem with my attempt is.

I am running Ubuntu 16.04 LTS Xenial Xerus, Linux kernel 4.4.0-57-generic.

Update #1

Ok, when running the command in "non background mode (-B)", I see an error that did not showed up before:

# btrfs replace start -B 2 /dev/sdd /mnt/brtfs-raid1-b                                                                                                                     
ERROR: ioctl(DEV_REPLACE_START) failed on "/mnt/brtfs-raid1-b": Read-only file system

/mnt/brtfs-raid1-b is mounted RO (Read Only). I have no choice; Btrfs does not allow me to mount the remaining disk as RW (Read Write). When I try to mount the disk RW, I get the following error in syslog:

BTRFS: missing devices(1) exceeds the limit(0), writeable mount is not allowed

When in RO mode, it seams I cannot do anything; cannot replace, nor add, nor delete a disk. But there is no way for me to mount the disk as RW. What option is left?

It shouldn't be this complicated when a simple disk fails. The system should continue running RW and warn me of a failed drive. I should be able to insert a new disk and have the data recopied over it, while the applications remain unaware of the disk issue. That is a proper RAID.

Answer 1

Update: According to @mkudlacek, this problem has been fixed.

For prosperity, here is my answer to why in 2017, I could not rebuild a RAID with a missing drive.

Turns out that this is a limitation of btrfs as of beginning of 2017. To get the filesystem mounted rw again, one needs to patch the kernel. I have not tried it though. I am planing to move away from btrfs because of this; one should not have to patch a kernel to be able to replace a faulty disk.

Click on the following links for details:

• Kernel patch here
• Full email thread

Please leave a comment if you still suffer from this problem as of 2020. I believe that people would like to know if this has been fixed or not.

Update: I moved to good old mdadm and lvm and am very happy with my RAID10 4x4 Tb (8 Tb total space), as of 2020-10-20. It is proven, works well, not resource intensive and I have full trust in it.

Answer 2

replace needs the filesystem to be mounted rw to operate.

In a degraded BTRFS RAID1 filesytem, you have one and only one chance to mount the filesystem rw using -o degraded:

   degraded
       (default: off)

       Allow mounts with less devices than the RAID profile constraints
       require. A read-write mount (or remount) may fail when there are
       too many devices missing, for example if a stripe member is
       completely missing from RAID0.

After rw mount, find the devid of the missing device:

btrfs filesystem show /mountpoint

Replace the missing with the new device:

btrfs replace start -B <devid> /dev/new-disk /mountpoint

Check the status:

btrfs replace status /mountpoint

replace will resume on reboot.

Answer 3

Add the new drive to the filesystem with btrfs device add /dev/sdd /mountpoint then remove the missing drive with btrfs dev del missing /mountpoint remounting the filesystem may be required before btrfs dev del missing will work.

Answer 4

btrfs replace is indeed the thing to try, but there are two gotchas regarding its invocation: it will only show errors when you use -B (otherwise it'll exit with status 0, as if everything is fine, but you'll see "never started" when you check the status), and invalid parameters will throw unrelated errors.

For example, I think my disk is fine but the RAID1 got out of sync somehow (probably a power outage during which the host survived, but the disks are not on backup power and might have come online at slightly different times). To check, when I power down disk B (while mounted), I can read data just fine. When I power down disk A instead (disk B is turned on, and the filesystem was already mounted) then I get errors and corrupt data. So clearly disk A is fine and disk B is corrupt. But disk B appears to function, so I want to re-use it and just rebuild. Therefore I want to replace /dev/diskB with /dev/diskB.

When I used btrfs replace start -B /dev/diskB /dev/diskB /mnt/btrfs it showed me ERROR: ioctl(DEV_REPLACE_START) failed on "/mnt/btrfs": Invalid argument, <illegal result value>. So there is a problem with the mountpoint it seems, right? Nope, when I changed the first /dev/diskB to /dev/diskA, it just worked. The mistake was in the devices, not in the mountpoint.

Similarly, I see the first argument (2) is kind of weird. Perhaps the error is wrong and it would work with a device in place of the 2?

btrfs replace has two modes of operating: one where you use the broken device as first argument (after start -B or whatever), and a mode (if the first option is unavailable) where you use the working device to be copied from. In either case, the second argument is the disk you wish to use to rebuild with.

Whether the filesystem is mounted read-only or read-write, does not seem to matter. That's why I suspect it's rejecting your arguments and giving you a wrong error, rather than the error being correct.