Btrfs: New device added to RAID1 not being used

Question

The problem

I added a new drive to my RAID1 array, and while it's listed as part of the array, it is not being written to for new data (in spite of being the largest drive and empty). Also, sudo btrfs fi usage does show it as part of the array, but is not listed under the Data,RAID1 nor Metadata,RAID1 sections (see output below).

Although every guide I could find shows that I added a new device to an existing RAID1 array correctly, it's not.

Prologue

Those familiar with RAID but not necessary Btrfs' implementation of it, might understandably assume that you can't do RAID1 with an odd number of disks. But Btrfs isn't actually traditional "RAID1", it's something else. It's arguably more flexible, and takes any number of any sized disks >1, makes sure that each chunk is written to any two different disks (usually starting with the ones with most free space), and it's (supposedly) good. It explicitly supports using an odd number of disks. Here is an example question about that.

Steps

First, I started with a functioning, recently balanced 4-disk Btrfs RAID1 array. (With 1x4tb, 2x8tb, and 1x10tb drives.) All four disks are part of the data and metadata RAID1 array (see output below).

Then I added a new 10tb disk, via:

sudo btrfs device add -f /dev/sdj /mnt/btrfs/ba07
sudo btrfs balance start /mnt/btrfs/ba07  ### As later noted, I also ran this with RAID1 conversion filters.

After that, here's the output from sudo btrfs fi usage /mnt/btrfs/ba07 that seems to indicate that it isn't actually part of the array:

Overall:
    Device size:                  36.38TiB
    Device allocated:             17.19TiB
    Device unallocated:           19.19TiB
    Device missing:                  0.00B
    Used:                         16.73TiB
    Free (estimated):              9.82TiB      (min: 9.82TiB)
    Data ratio:                       2.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 96.53MiB)

Data,RAID1: Size:8.58TiB, Used:8.35TiB
   /dev/sdf        5.68TiB
   /dev/sdg        5.53TiB
   /dev/sdh        3.57TiB
   /dev/sdi        2.38TiB

Metadata,RAID1: Size:18.00GiB, Used:14.21GiB
   /dev/sdf        8.00GiB
   /dev/sdg       17.00GiB
   /dev/sdh        4.00GiB
   /dev/sdi        7.00GiB

System,RAID1: Size:32.00MiB, Used:1.47MiB
   /dev/sdh       32.00MiB
   /dev/sdj       32.00MiB

Unallocated:
   /dev/sdf        1.59TiB
   /dev/sdg        1.73TiB
   /dev/sdh        5.52TiB
   /dev/sdi        1.26TiB
   /dev/sdj        9.09TiB

Notice the entries for Data,RAID1 and Metadata,RAID1 don't include the new disk, /dev/sdj.

Troubleshooting steps

Shooting in the dark, I tried:

sudo btrfs filesystem resize max /mnt/btrfs/ba07

and

sudo btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt/btrfs/ba07

The results are the same, the disk is not being listed under Data,RAID1 or Metadata,RAID1.

More recently I tried the tips from the comment from EarthMind below, combined with this recommendation:

btrfs device delete /dev/sdj /mnt/btrfs/ba07              #Succeeded
btrfs device add    /dev/sdj /mnt/btrfs/ba07              #Succeeded
btrfs balance start -mconvert=raid1,soft /mnt/btrfs/ba07  #Tip said to do these balances separately, metadata first
btrfs balance start -dconvert=raid1,soft /mnt/btrfs/ba07

All steps finished immediately, and with nothing being done in the background; presumably this was because I had already run variations on balancing numerous times already, and also there was no data to on sdj to move before removing.

sudo btrfs fi usage showed the same thing: /dev/sdj not part of Data or Metadata RAID1.

Since the definition of insanity is popularly defined as doing the same thing over and over expecting different results, I'm currently doing just that:

sudo btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt/btrfs/ba07

For whatever reason, that seems to actually be doing something. (But not when done individually, and even though I've tried the exact same command several times earlier.) Specifically, /dev/sdj appears to be undergoing writes, which is the whole goal here. It's too early to claim victory, as this could take many hours to days to finish. Specifically, this part of the output from btrfs fi usage /mnt/btrfs/ba07 seems promising:

Data,RAID1: Size:8.40TiB, Used:8.36TiB
   /dev/sdf        5.56TiB
   /dev/sdg        5.45TiB
   /dev/sdh        3.47TiB
   /dev/sdi        2.31TiB
   /dev/sdj       19.00GiB

The last line, /dev/sdj, is growing.

Additional info

System info:

$ btrfs --version
btrfs-progs v4.15.1

$ uname -r
4.15.0-54-generic

$ lsb_release -d
Description:    Ubuntu 18.04.2 LTS

Similar problems on the web

• btrfs raid1 not using all disks?
  • Was behaving as expected. Since existing drives had more free space, they were being written to before the new disk. My situation is the opposite, new drive has more free space.
• Btrfs adding smaller drive to RAID1
  • User wound up having to take a different approach altogether when a disk wound up dying.
• Why does btrfs refuse to convert metadata to RAID1
  • Was due to a regression in the specific Btrfs version (4.0). But I tried these steps on both 4.4, and 4.15.
• BTRFS balance completed, but still shows data stored in “single” mode
  • Suggested adding "soft" parameter to convert filters, and running -mconvert=raid1,soft separately before -dconvert=raid1,soft. This solution didn't solve my problem.

Summary

Clearly I must be doing something wrong, but hours of googling and experimenting have so far yielded nothing, other than "it was done correctly and 'should' work".

It tentatively seems to be working with sudo btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt/btrfs/ba07, after having removed and re-added the disk from the array, even though that's the third or fourth try with the same command. I'll add this as an answer in a few hours or days, whenever it finishes:

#Repeat the third command below a few times if the whole thing doesn't work the first time.
#Note, don't specify "soft" parameter.
#If it's still not working, try the whole 3-line series of commands below multiple times.
#Because, software.

btrfs device delete /dev/sdj /mnt/btrfs/ba07
btrfs device add    /dev/sdj /mnt/btrfs/ba07
sudo btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt/btrfs/ba07

Answer 1

I figured out how to fix it in this particular case (but not why it was happening). Nor can I be sure it would have "fixed" similar issues noted toward the end of the question above.

Either way, this clearly seems to be undesired, unintended, or at best undocumented behavior on Btrfs' part. (For example, it may just be "undocumented" that you have to do these steps in order for Btrfs to start using a newly added device. As to why it doesn't always work...is another question.)

In this (real) example:

• /dev/sdj is the device to add, that btrfs isn't using to distribute data to.
• /mnt/btrfs/ba07 is the mounted location for the array.

btrfs device delete /dev/sdj /mnt/btrfs/ba07
btrfs device add    /dev/sdj /mnt/btrfs/ba07
sudo btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt/btrfs/ba07

Usual disclaimers, make sure your data is backed up. The device delete command is supposed to do so without data loss (and did), and should work as long as there's enough room on the other disks to receive the removed device's data (which is obviously the case if it's not even being used), but better safe than sorry.

If it didn't work, and assuming the rest of your array does have data and the newly added disk is empty, you'll notice after the btrfs balance start... that:

• The command (btrfs balance start...) probably returns right away,
• btrfs balance status /mountpoint shows no activity, and
• The Data,RAID1 section of btrfs fi usage output, does not list the new device.

If it didn't work, try the last command (btrfs balance start...) again.

If that still doesn't work, try all three commands again.

I don't know if doing it multiple times is always required, or why it necessary in this case. (In my case, the last command by itself didn't work no matter how many times it was run. But after deleting and re-adding the device, then doing the balance command twice (with some elapsed time and random unrelated commands in between), it started working the second time.

Also note that in this particular case, these didn't work, even after multiple attempts:

• Performing the metadata balance separately, before the data balance (as was suggested in a similar question, documented in the question above).
• Adding the ,soft parameter to the -mconvert=raid1 and -dconvert=raid1 arguments.