3

I have an existing 6x16TB BTRFS raid10 and wanted to know if I can extend with 2 or 4 more 16TB HDDs?

I know this is not possible with traditional software/fake/hardware raids, but BTRFS gives me hope with its rebalance and add features.

  • Is that possible?
  • What are commands to achieve that? And,
  • will I keep the 1 drive failure feature after eventual resize?
2
  • changed your title a bit; I think you're actually interested in how to do that if it's possible than your title let on :) Jan 18, 2023 at 8:45
  • Thank you! Looks more structured now :)
    – tio78
    Jan 18, 2023 at 11:18

1 Answer 1

4

Okay. Just brought up an ubuntu 22.04 VM with 6x1GB virtual HDDs. Added 4 of them as a start:

### Created the raid 4 drives
root@ubuntu-for-devops:~# mkfs.btrfs -L data -d raid10 -m raid10 -f /dev/sdc /dev/sdd /dev/sde /dev/sdf

### mounted the raid10
root@ubuntu-for-devops:~# mount /dev/disk/by-label/data /mnt/

### checked space and created a random 100MB file on the raid10
root@ubuntu-for-devops:~# df -hT
Filesystem                  Type   Size  Used Avail Use% Mounted on
/dev/sdc                    btrfs  2.0G  104M  1.8G   6% /mnt
root@ubuntu-for-devops:~# dd if=/dev/urandom of=/mnt/somefile bs=1M count=100

### Added two more devices 1GB each
root@ubuntu-for-devops:~# btrfs device add /dev/sda /dev/sdg /mnt/

### Turned out they are not part of the RAID yet, though space grew from 2GB to 3GB (I think it should be 4GB as we added 2x1GB)
root@ubuntu-for-devops:~# df -hT
Filesystem                 Type   Size  Used Avail Use% Mounted on
/dev/sdc                   btrfs  3.0G  104M  2.8G   4% /mnt
root@ubuntu-for-devops:~# btrfs device usage /mnt/
/dev/sdc, ID: 1
   Device size:             1.00GiB
   Device slack:              0.00B
   Data,RAID10/4:         102.38MiB
   Metadata,RAID10/4:      64.00MiB
   System,RAID10/4:         8.00MiB
   Unallocated:           849.62MiB

/dev/sdd, ID: 2
   Device size:             1.00GiB
   Device slack:              0.00B
   Data,RAID10/4:         102.38MiB
   Metadata,RAID10/4:      64.00MiB
   System,RAID10/4:         8.00MiB
   Unallocated:           849.62MiB

/dev/sde, ID: 3
   Device size:             1.00GiB
   Device slack:              0.00B
   Data,RAID10/4:         102.38MiB
   Metadata,RAID10/4:      64.00MiB
   System,RAID10/4:         8.00MiB
   Unallocated:           849.62MiB

/dev/sdf, ID: 4
   Device size:             1.00GiB
   Device slack:              0.00B
   Data,RAID10/4:         102.38MiB
   Metadata,RAID10/4:      64.00MiB
   System,RAID10/4:         8.00MiB
   Unallocated:           849.62MiB

/dev/sda, ID: 5
   Device size:             1.00GiB
   Device slack:              0.00B
   Unallocated:             1.00GiB

/dev/sdg, ID: 6
   Device size:             1.00GiB
   Device slack:              0.00B
   Unallocated:             1.00GiB

### Had to rebalance the raid10's metadata and data
root@ubuntu-for-devops:~# btrfs balance start -dconvert=raid10 -mconvert=raid10 /mnt
Done, had to relocate 3 out of 3 chunks
root@ubuntu-for-devops:~# btrfs device usage /mnt/
/dev/sdc, ID: 1
   Device size:             1.00GiB
   Device slack:              0.00B
   Data,RAID10/6:         208.00MiB
   Metadata,RAID10/6:      96.00MiB
   System,RAID10/6:        32.00MiB
   Unallocated:           688.00MiB

/dev/sdd, ID: 2
   Device size:             1.00GiB
   Device slack:              0.00B
   Data,RAID10/6:         208.00MiB
   Metadata,RAID10/6:      96.00MiB
   System,RAID10/6:        32.00MiB
   Unallocated:           688.00MiB

/dev/sde, ID: 3
   Device size:             1.00GiB
   Device slack:              0.00B
   Data,RAID10/6:         208.00MiB
   Metadata,RAID10/6:      96.00MiB
   System,RAID10/6:        32.00MiB
   Unallocated:           688.00MiB

/dev/sdf, ID: 4
   Device size:             1.00GiB
   Device slack:              0.00B
   Data,RAID10/6:         208.00MiB
   Metadata,RAID10/6:      96.00MiB
   System,RAID10/6:        32.00MiB
   Unallocated:           688.00MiB

/dev/sda, ID: 5
   Device size:             1.00GiB
   Device slack:              0.00B
   Data,RAID10/6:         208.00MiB
   Metadata,RAID10/6:      96.00MiB
   System,RAID10/6:        32.00MiB
   Unallocated:           688.00MiB

/dev/sdg, ID: 6
   Device size:             1.00GiB
   Device slack:              0.00B
   Data,RAID10/6:         208.00MiB
   Metadata,RAID10/6:      96.00MiB
   System,RAID10/6:        32.00MiB
   Unallocated:           688.00MiB

### We lost another 100+MB because of the new metadata
root@ubuntu-for-devops:~# df -hT
Filesystem                 Type   Size  Used Avail Use% Mounted on
/dev/sdc                   btrfs  3.0G  104M  2.6G   4% /mnt

### Time to check if the raid will sustain a one drive failure by erasing sdd completely
root@ubuntu-for-devops:~# dd if=/dev/urandom of=/dev/sdd bs=1M conv=fsync status=progress

### After simulating a write the dmesg reflected the issue:
root@ubuntu-for-devops:~# touch /mnt/123

root@ubuntu-for-devops:~# dmesg
...
[ 2198.260580] BTRFS warning (device sdc): csum failed root 5 ino 257 off 131072 csum 0x35b08b64 expected csum 0xbae6547e mirror 1
[ 2198.260626] BTRFS error (device sdc): bdev /dev/sdd errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[ 2198.260895] BTRFS warning (device sdc): csum failed root 5 ino 257 off 135168 csum 0x343bf823 expected csum 0xb1298672 mirror 1
[ 2198.260901] BTRFS error (device sdc): bdev /dev/sdd errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[ 2198.263315] BTRFS warning (device sdc): csum failed root 5 ino 257 off 139264 csum 0x654231d1 expected csum 0x6f03029b mirror 1
[ 2198.263322] BTRFS error (device sdc): bdev /dev/sdd errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
[ 2198.263547] BTRFS warning (device sdc): csum failed root 5 ino 257 off 143360 csum 0xa9f20424 expected csum 0xac791696 mirror 1
[ 2198.263553] BTRFS error (device sdc): bdev /dev/sdd errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
[ 2198.263791] BTRFS warning (device sdc): csum failed root 5 ino 257 off 147456 csum 0x224092d4 expected csum 0x7bba1416 mirror 1
[ 2198.263797] BTRFS error (device sdc): bdev /dev/sdd errs: wr 0, rd 0, flush 0, corrupt 5, gen 0
[ 2198.264935] BTRFS warning (device sdc): csum failed root 5 ino 257 off 151552 csum 0xfc08be81 expected csum 0x221b75d8 mirror 1
[ 2198.264940] BTRFS error (device sdc): bdev /dev/sdd errs: wr 0, rd 0, flush 0, corrupt 6, gen 0
[ 2198.265129] BTRFS warning (device sdc): csum failed root 5 ino 257 off 155648 csum 0xfa1929bb expected csum 0xedc93828 mirror 1
[ 2198.265154] BTRFS error (device sdc): bdev /dev/sdd errs: wr 0, rd 0, flush 0, corrupt 7, gen 0
[ 2198.265993] BTRFS warning (device sdc): csum failed root 5 ino 257 off 159744 csum 0xb7c7453d expected csum 0xfef92d74 mirror 1
[ 2198.265998] BTRFS error (device sdc): bdev /dev/sdd errs: wr 0, rd 0, flush 0, corrupt 8, gen 0
[ 2198.266201] BTRFS warning (device sdc): csum failed root 5 ino 257 off 163840 csum 0x37c8083f expected csum 0x56e50bbd mirror 1
[ 2198.266204] BTRFS error (device sdc): bdev /dev/sdd errs: wr 0, rd 0, flush 0, corrupt 9, gen 0
[ 2198.267108] BTRFS warning (device sdc): csum failed root 5 ino 257 off 167936 csum 0x7b5bbbe0 expected csum 0x415c3cf5 mirror 1
[ 2198.267112] BTRFS error (device sdc): bdev /dev/sdd errs: wr 0, rd 0, flush 0, corrupt 10, gen 0
[ 2198.271391] BTRFS info (device sdc): read error corrected: ino 257 off 131072 (dev /dev/sdd sector 359168)
[ 2198.271855] BTRFS info (device sdc): read error corrected: ino 257 off 135168 (dev /dev/sdd sector 359176)
[ 2198.272376] BTRFS info (device sdc): read error corrected: ino 257 off 139264 (dev /dev/sdd sector 359184)

### Time for scrub data testing
root@ubuntu-for-devops:/mnt# btrfs scrub start /mnt/
scrub started on /mnt/, fsid 8d21c38c-9697-4998-b5a7-73858939e7dd (pid=2002)
root@ubuntu-for-devops:/mnt# WARNING: errors detected during scrubbing, corrected

root@ubuntu-for-devops:/mnt# btrfs scrub status /mnt/
UUID:             8d21c38c-9697-4998-b5a7-73858939e7dd
Scrub started:    Wed Jan 18 12:30:14 2023
Status:           finished
Duration:         0:00:00
Total to scrub:   200.50MiB
Rate:             0.00B/s
Error summary:    csum=1
  Corrected:      1
  Uncorrectable:  0
  Unverified:     0
...

It looks like BTRFS is fine with growing an existing raid10 and can sustain a single drive failure, as expected.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .