1

I have a Btrfs raid1 with 3 disks on a Ubuntu 16.04. However, it seems only 2 disks are being used instead of all 3. How should I fix this?

root@one:~# btrfs fi sh
Label: none  uuid: 3880b9fa-0824-4ffe-8f61-893a104f3567
            Total devices 3 FS bytes used 54.77GiB
            devid    1 size 2.73TiB used 56.03GiB path /dev/sda2
            devid    2 size 2.73TiB used 56.03GiB path /dev/sdc2
            devid    3 size 2.59TiB used 0.00B path /dev/sdb3

I have tried running a conversion filter but still the /dev/sdb3 is not being used.

root@one:~# btrfs balance start -dconvert=raid1 -mconvert=raid1 /top/raid/
Done, had to relocate 112 out of 112 chunks

root@one:~# btrfs fi df /top/raid/
Data, RAID1: total=55.00GiB, used=54.40GiB
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=1.00GiB, used=373.06MiB
GlobalReserve, single: total=128.00MiB, used=0.00B

At first, there's only 1 disk during Ubuntu server installation. Then I added a disk and converted to raid1. Then I added a thrid disk /dev/sdb3 and tried to balance again. The third disk is not being used.

root@one:~# btrfs --version
btrfs-progs v4.4

I can mount /dev/sdb3 just fine.

root@one:~# mount /dev/sdb3 /mnt
root@one:~# ll /mnt
total 16
drwxr-xr-x 1 root    root     74 Apr 13 09:37 ./
drwxr-xr-x 1 root    root    200 Apr 12 21:19 ../
drwxr-xr-x 1 root    root    200 Apr 12 21:19 @/
drwxr-xr-x 1 root    root    152 Apr 12 15:31 @home/
drwxrwx--t 1 root    root     36 Apr 13 09:38 @samba/

root@one:~# btr fi sh
Label: none  uuid: 3880b9fa-0824-4ffe-8f61-893a104f3567
        Total devices 3 FS bytes used 54.82GiB
        devid    1 size 2.73TiB used 56.03GiB path /dev/sda2
        devid    2 size 2.73TiB used 56.03GiB path /dev/sdc2
        devid    3 size 2.59TiB used 0.00B path /dev/sdb3
  • 2
    it's my understanding that btrfs raid1 on > 2 devices still only creates two copies (citation needed) – Jeff Schaller Apr 13 '17 at 12:30
  • related, similar: unix.stackexchange.com/q/281255/117549 – Jeff Schaller Apr 13 '17 at 12:34
  • I don't know how exactly the allocation process works, but playing around with btrfs over different devices I noticed it usually uses those with the most free space. In the example of the question I think it will start using all 3 disks after you allocate about 140GB of data (2.73-2.59=0.14). Just try putting 200GB of data and see what happens. – Falk Jan 15 '18 at 11:06
1

EDIT:

NOTE: The btrfs FAQ states the following, as commented by @jeff-schaller (emphasis mine):

btrfs supports RAID-0, RAID-1, and RAID-10. As of Linux 3.9, btrfs also supports RAID-5 and RAID-6 although that code is still experimental.

btrfs combines all the devices into a storage pool first, and then duplicates the chunks as file data is created. RAID-1 is defined currently as "2 copies of all the data on different devices". This differs from MD-RAID and dmraid, in that those make exactly n copies for n devices. In a btrfs RAID-1 on three 1 TB devices we get 1.5 TB of usable data. Because each block is only copied to 2 devices, writing a given block only requires exactly 2 devices to be written to; reading can be made from only one.

RAID-0 is similarly defined, with the stripe split across as many devices as possible. 3 × 1 TB devices yield 3 TB usable space, but offers no redundancy at all.

RAID-10 is built on top of these definitions. Every stripe is split across to exactly 2 RAID-1 sets and those RAID-1 sets are written to exactly 2 devices (hence 4 devices minimum). A btrfs RAID-10 volume with 6 × 1 TB devices will yield 3 TB usable space with 2 copies of all data.

I do not have large enough drives on hand to test this at the moment, but my speculation is simply that, since you have relatively large drives, btrfs simply chose to write the data to the first two drives thus far. I would expect that to change in the future as more data is written to the drives.

In case you are interested in my tests with smaller drives:

I installed Ubuntu Server 16.04 LTS in a VM with a single SATA drive, installed the OS on a single btrfs partition.

Then I added another SATA drive, partitioned it, ran btrfs device add /dev/sdb1 /, and then balanced it while converting to raid1 with btrfs balance start -dconvert=raid1 -mconvert=raid1 /

I repeated for device /dev/sdc1. The result for me is the same - I have a btrfs spanning three drives. I also fallocated a 2GiB file, and it was indeed accessible from all three disks. My btrfs fi sh shows the following:

Label: none  uuid: cdfe192c-36da-4a3c-bc1a-74137abbb190
    Total devices 3 FS bytes used 3.07GiB
    devid    1 size 10.00GiB used 5.25GiB path /dev/sda1
    devid    2 size 10.00GiB used 5.03GiB path /dev/sdb1
    devid    3 size 8.00GiB used 2.28GiB path /dev/sdc1

How did you call mkfs.btrfs? What is your btrfs-progs version?

# btrfs --version
btrfs-progs v4.4

I cannot reproduce your situation. What happens if you try to mount /dev/sdb3?

If you have a virtual machine or a spare disk to play with partitioning, create 3 partitions and try the following.

I created an Ubuntu 16.04 VM and partitioned /dev/vda into three partitions of 2GiB each.

# mkfs.btrfs -d raid1 -m raid1 /dev/vda{1..3}

Label:              (null)
UUID:               0d6278f7-8830-4a73-a72f-0069cc560aaf
Node size:          16384
Sector size:        4096
Filesystem size:    6.00GiB
Block group profiles:
  Data:             RAID1           315.12MiB
  Metadata:         RAID1           315.12MiB
  System:           RAID1            12.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Number of devices:  3
Devices:
   ID        SIZE  PATH
    1     2.00GiB  /dev/vda1
    2     2.00GiB  /dev/vda2
    3     2.00GiB  /dev/vda3

# btrfs fi sh

Label: none  uuid: 0d6278f7-8830-4a73-a72f-0069cc560aaf
    Total devices 3 FS bytes used 112.00KiB
    devid    1 size 2.00GiB used 614.25MiB path /dev/vda1
    devid    2 size 2.00GiB used 315.12MiB path /dev/vda2
    devid    3 size 2.00GiB used 315.12MiB path /dev/vda3

Try mounting /dev/vda1, writing a file to it, then mounting /dev/vda2 or /dev/vda3 instead and checking if the file is there (It definitely should be).

PS: I first tried this on Arch with btrfs-progs version 4.10.2 with the same results, but thought that probably Ubuntu 16.04 ships with an older version that might behave differently. Turns out it ships with v4.4, but it seems to behave the same in regards to filesystem creation and mirroring etc.

  • At first, there's only 1 disk during Ubuntu server installation. Then I added a disk and converted to raid1. Then I added the thrid disk /dev/sdb3 and tried to balance again. The third disk is not being used. <code> # btrfs --version btrfs-progs v4.4 I can mount /dev/sdb3 just fine. # mount /dev/sdb3 /mnt # ll /mnt total 16 drwxr-xr-x 1 root root 74 Apr 13 09:37 ./ drwxr-xr-x 1 root root 200 Apr 12 21:19 ../ drwxr-xr-x 1 root root 200 Apr 12 21:19 @/ drwxr-xr-x 1 root root 152 Apr 12 15:31 @home/ drwxrwx--t 1 root root 36 Apr 13 09:38 @samba/ – tkittich Apr 13 '17 at 14:12
0

The volume is now using all the disks. I guess Rabban's speculation should be correct. Since the first two disks are bigger than the third, "btrfs simply chose to write the data to the first two drives thus far".

root@one:~# btrfs fi sh
Label: none  uuid: 3880b9fa-0824-4ffe-8f61-893a104f3567
        Total devices 3 FS bytes used 151.00GiB
        devid    1 size 2.73TiB used 148.03GiB path /dev/sda2
        devid    2 size 2.73TiB used 148.03GiB path /dev/sdc2
        devid    3 size 2.59TiB used 8.00GiB path /dev/sdb3

root@one:~# btrfs dev usage /
/dev/sda2, ID: 1
   Device size:             2.73TiB
   Data,RAID1:            147.00GiB
   Metadata,RAID1:          1.00GiB
   System,RAID1:           32.00MiB
   Unallocated:             2.58TiB

/dev/sdb3, ID: 3
   Device size:             2.59TiB
   Data,RAID1:              8.00GiB
   Unallocated:             2.58TiB

/dev/sdc2, ID: 2
   Device size:             2.73TiB
   Data,RAID1:            147.00GiB
   Metadata,RAID1:          1.00GiB
   System,RAID1:           32.00MiB
   Unallocated:             2.58TiB

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.