The problem
I added a new drive to my RAID1 array, and while it's listed as part of the array, it is not being written to for new data (in spite of being the largest drive and empty). Also, sudo btrfs fi usage
does show it as part of the array, but is not listed under the Data,RAID1
nor Metadata,RAID1
sections (see output below).
Although every guide I could find shows that I added a new device to an existing RAID1 array correctly, it's not.
Prologue
Those familiar with RAID but not necessary Btrfs' implementation of it, might understandably assume that you can't do RAID1 with an odd number of disks. But Btrfs isn't actually traditional "RAID1", it's something else. It's arguably more flexible, and takes any number of any sized disks >1, makes sure that each chunk is written to any two different disks (usually starting with the ones with most free space), and it's (supposedly) good. It explicitly supports using an odd number of disks. Here is an example question about that.
Steps
First, I started with a functioning, recently balanced 4-disk Btrfs RAID1 array. (With 1x4tb, 2x8tb, and 1x10tb drives.) All four disks are part of the data and metadata RAID1 array (see output below).
Then I added a new 10tb disk, via:
sudo btrfs device add -f /dev/sdj /mnt/btrfs/ba07
sudo btrfs balance start /mnt/btrfs/ba07 ### As later noted, I also ran this with RAID1 conversion filters.
After that, here's the output from sudo btrfs fi usage /mnt/btrfs/ba07
that seems to indicate that it isn't actually part of the array:
Overall:
Device size: 36.38TiB
Device allocated: 17.19TiB
Device unallocated: 19.19TiB
Device missing: 0.00B
Used: 16.73TiB
Free (estimated): 9.82TiB (min: 9.82TiB)
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 96.53MiB)
Data,RAID1: Size:8.58TiB, Used:8.35TiB
/dev/sdf 5.68TiB
/dev/sdg 5.53TiB
/dev/sdh 3.57TiB
/dev/sdi 2.38TiB
Metadata,RAID1: Size:18.00GiB, Used:14.21GiB
/dev/sdf 8.00GiB
/dev/sdg 17.00GiB
/dev/sdh 4.00GiB
/dev/sdi 7.00GiB
System,RAID1: Size:32.00MiB, Used:1.47MiB
/dev/sdh 32.00MiB
/dev/sdj 32.00MiB
Unallocated:
/dev/sdf 1.59TiB
/dev/sdg 1.73TiB
/dev/sdh 5.52TiB
/dev/sdi 1.26TiB
/dev/sdj 9.09TiB
Notice the entries for Data,RAID1
and Metadata,RAID1
don't include the new disk, /dev/sdj
.
Troubleshooting steps
Shooting in the dark, I tried:
sudo btrfs filesystem resize max /mnt/btrfs/ba07
and
sudo btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt/btrfs/ba07
The results are the same, the disk is not being listed under Data,RAID1
or Metadata,RAID1
.
More recently I tried the tips from the comment from EarthMind below, combined with this recommendation:
btrfs device delete /dev/sdj /mnt/btrfs/ba07 #Succeeded
btrfs device add /dev/sdj /mnt/btrfs/ba07 #Succeeded
btrfs balance start -mconvert=raid1,soft /mnt/btrfs/ba07 #Tip said to do these balances separately, metadata first
btrfs balance start -dconvert=raid1,soft /mnt/btrfs/ba07
All steps finished immediately, and with nothing being done in the background; presumably this was because I had already run variations on balancing numerous times already, and also there was no data to on sdj to move before removing.
sudo btrfs fi usage
showed the same thing: /dev/sdj not part of Data or Metadata RAID1.
Since the definition of insanity is popularly defined as doing the same thing over and over expecting different results, I'm currently doing just that:
sudo btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt/btrfs/ba07
For whatever reason, that seems to actually be doing something. (But not when done individually, and even though I've tried the exact same command several times earlier.) Specifically, /dev/sdj appears to be undergoing writes, which is the whole goal here. It's too early to claim victory, as this could take many hours to days to finish. Specifically, this part of the output from btrfs fi usage /mnt/btrfs/ba07
seems promising:
Data,RAID1: Size:8.40TiB, Used:8.36TiB
/dev/sdf 5.56TiB
/dev/sdg 5.45TiB
/dev/sdh 3.47TiB
/dev/sdi 2.31TiB
/dev/sdj 19.00GiB
The last line, /dev/sdj, is growing.
Additional info
System info:
$ btrfs --version
btrfs-progs v4.15.1
$ uname -r
4.15.0-54-generic
$ lsb_release -d
Description: Ubuntu 18.04.2 LTS
Similar problems on the web
- btrfs raid1 not using all disks?
- Was behaving as expected. Since existing drives had more free space, they were being written to before the new disk. My situation is the opposite, new drive has more free space.
- Btrfs adding smaller drive to RAID1
- User wound up having to take a different approach altogether when a disk wound up dying.
- Why does btrfs refuse to convert metadata to RAID1
- Was due to a regression in the specific Btrfs version (4.0). But I tried these steps on both 4.4, and 4.15.
- BTRFS balance completed, but still shows data stored in “single” mode
- Suggested adding "soft" parameter to convert filters, and running
-mconvert=raid1,soft
separately before-dconvert=raid1,soft
. This solution didn't solve my problem.
- Suggested adding "soft" parameter to convert filters, and running
Summary
Clearly I must be doing something wrong, but hours of googling and experimenting have so far yielded nothing, other than "it was done correctly and 'should' work".
It tentatively seems to be working with sudo btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt/btrfs/ba07
, after having removed and re-added the disk from the array, even though that's the third or fourth try with the same command. I'll add this as an answer in a few hours or days, whenever it finishes:
#Repeat the third command below a few times if the whole thing doesn't work the first time.
#Note, don't specify "soft" parameter.
#If it's still not working, try the whole 3-line series of commands below multiple times.
#Because, software.
btrfs device delete /dev/sdj /mnt/btrfs/ba07
btrfs device add /dev/sdj /mnt/btrfs/ba07
sudo btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt/btrfs/ba07
sudo btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt/btrfs/ba07
instead ofsudo btrfs balance start /mnt/btrfs/ba07
? I think a balance by default doesn't including metadata and data necessarilysudo btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt/btrfs/ba07
(as noted later in the question), but did not try removing the device first. That's a good idea, I'll try it, thanks.