JBOD: How to Raid1 drives with different capacities? Btrfs, ZFS?

Question

I have selection of hard drives, with capacities:

1 TB
500 GB
320 GB
250 GB

I'd like to make Raid1 out of them, in the sense, that copy of any data will be always on two drives out of four I have. Problem is, that all drives have different capacities. In the past I've been running normal Raid1 mdadm 2x 2TB, Btrfs with 2x 2TB after I migrated from mdadm. I don't mind to use mdadm again, or Btrfs, or ZFS, although I know nothing about that one.

What would be best filesystem or utility for me, which fits this criteria:

1. to have two copies of files at all times, so if one disk dies, I don't lose data
2. have at much at possible available space with still meet criteria from point 1

What should I use? I have feeling Btrfs Raid1 will handle this, and story exactly two copies of every file across all four hard drives, but I am not sure. I have Linux on AMD64 computer, with 5.10 kernel.

Any advice welcome.

EDIT - SOLUTION!

Thanks very much for all your replies! So I went ahead with Btrfs Raid1 configuration, It was time consuming process, but very painless, because I'm already familiar Btrfs and use it daily. I figured that wouldn't lose any physical space with Raid1, while maintaining redundancy ratio of 2. I used this calculator: https://carfax.org.uk/btrfs-usage/

My device sizes:

931.51 GiB
465.76 GiB
298.09 GiB
232.89 GiB

Total space for files in Raid1 mode: 963 GiB (all space used)
Total space for files in Raid5 mode: 995 GiB (466 GiB unusable on biggest drive)

With Raid1, I have now 719.5 GiB data stored and 241.6 GiB free, for total of 961.1 GiB, which matches what calculator said. And I can add/remove drives later at will. Amazing.

I'm tempted to go for Raid5, because it gives theoretical read speed gain of 4x, while Raid1 gives 2x. But I heard it's a bit experimental at the moment and not recommended for daily use, it that correct? What's its state on 5.10 kernel, anyone uses it and can share their experience?

Following building the array, adding all drives one by one and migrating all data to it (I have backups), I did full scrub and everything is fine:

$ sudo btrfs scrub start -Bd /home

Scrub device /dev/sda (id 1) done
Scrub started:    Thu Jun 17 10:10:38 2021
Status:           finished
Duration:         2:38:51
Total to scrub:   724.03GiB
Rate:             77.53MiB/s
Error summary:    no errors found

Scrub device /dev/sdd (id 2) done
Scrub started:    Thu Jun 17 10:10:38 2021
Status:           finished
Duration:         1:52:39
Total to scrub:   374.03GiB
Rate:             56.57MiB/s
Error summary:    no errors found

Scrub device /dev/sdc (id 3) done
Scrub started:    Thu Jun 17 10:10:38 2021
Status:           finished
Duration:         0:48:06
Total to scrub:   207.00GiB
Rate:             73.09MiB/s
Error summary:    no errors found

Scrub device /dev/sdb (id 4) done
Scrub started:    Thu Jun 17 10:10:38 2021
Status:           finished
Duration:         0:30:00
Total to scrub:   143.00GiB
Rate:             80.89MiB/s
Error summary:    no errors found

Very happy!

EDIT - COMMENTS FOR @Marcus Müller:

"Remarks: are you sure you want this? 320 GB and 250 GB are probably not very new drives (...)"

You are very correct, these are very old drives! I removed my current 2x 2TB Seagate Barracuda Compute drives which I had on Btrfs Raid1 for last few years, to put these random 4x old aged drives. Barracudas are going to my server, which needs total reliability, I have 2x SSD 250GB for / there (Ext4 mdadm), and 2x HDD TB for /home and /var (Btrfs Raid1), and with ECC RAM. Another two Barracudas will join first two, for Btrfs Raid1 mode or possibly Raid10 or Raid5.

4x Old drives are now sitting in my desktop, they consist of:

Western Digital Blue Mobile, WDC WD10JPVX-60JC3T0
[1.00 TB, 4K sectors, SATA 3.0, 6.0 Gb/s, 5400 RPM, 2.5"]
Power_On_Hours 2496
Reallocated_Sector_Ct 0

Toshiba 2.5" HDD MK..65GSX, TOSHIBA MK5065GSX
[500 GB, 512 sectors, SATA 2.6, 3.0 Gb/s, 5400 RPM, 2.5"]
Power_On_Hours 2519
Reallocated_Sector_Ct 293

Seagate Momentus Thin, ST320LT007-9ZV142
[320 GB, 4K sectors, SATA 2.6, 3.0 Gb/s, 7200 RPM, 2.5"]
Power_On_Hours 5152
Reallocated_Sector_Ct 0

Seagate Barracuda 7200.12,ST3250318AS
[250 GB, 512 sectors, SATA 2.6, 3.0 Gb/s, 7200 RPM, 3.5"]
Power_On_Hours 6591
Reallocated_Sector_Ct 3251

That's a monster cocktail, I have to admit :D haha. Reliability is not an issue really, I have backups, but I don't want any painful surprises too quickly, so basic redundancy of 2.0 will save my trouble if one of the drives die within a year or so, I will just slap in another one or two.

Disclaimer: I Spinrite'd all drives (I do that on regular basis on all my drives), zero'd them, benchmark them, fully scrubbed Btrfs filesystem (I also do that on regular basis), they have no further bad sectors at the moment. And if they develop more over time, that would be perfect test for Spinrite to fix them, or at least for me to learn what Btrfs will do in this scenario!

Answer 1

Use btrfs because, unlike ZFS, it can efficiently handle drives of different sizes. You'd end up with roughly the total capacity of all drives divided by two (with 2 copies for redundancy). Approx 1TB. Not counting transparent compression.

ZFS would just create a vdev with a capacity of the smallest device - but with 4 drives, you'd make two mirror vdevs. e.g. a mirror-1 vdev with the 1TB and 500GB and a mirror-2 vdev with the 320GB and 250GB. To minimise wasted space, you can partition the 1TB and 320GB drives so that they have a 500GB partition and a 250GB partition to match their mirror drives. That would give the pool a total of 750GB capacity (in a RAID1+0 -like configuration) before compression. The remaining space (500GB and 70GB) on those drives can be used for something else, like stuff you don't care about because it's downloaded from and "backed up on the internet".

If you can afford it, I recommend replacing the 250GB drive with another 1TB drive. That would be great for zfs and pretty good for btrfs, as it would increase capacity to approx 1.3TB for either of them, before transparent compression. With much less "wasted" space.

Don't use LVM or mdadm to do the RAID-1/mirroring, btrfs can do that itself - and it would defeat much of the purpose of using an error correcting filesystem like btrfs.

Features like snapshots and transparent compression would still work, but you'd lose the ability for either of those filesystems to correct errors, because you're effectively giving it just a single drive.

You can use software or hardware raid under either ZFS or btrfs if you want to but a) it's not needed, and b) it's not a good idea, in fact it's a terrible idea. It's more work and more complex to set up, removes a vital feature, and provides no compensating benefit.

If you want to use LVM or mdadm, use ext4 (or, better yet, xfs) on top of them.

BTW, I still don't think I'd trust btrfs for RAID-5/6 but it's absolutely fine for RAID-1 or RAID-10 like mirrors.

Answer 2

I'd simply have one LVM physical volume on each of these drives:

pvcreate /dev/drive1 /dev/drive2 /dev/drive3 /dev/drive4

then create a Volume Group out of these:

vgcreate myvolumegroup /dev/drive1 /dev/drive2 /dev/drive3 /dev/drive4

Finally, create a logical volume on that has the property of every block having one mirror:

lvcreate --mirrors 1 -l100%FREE  -n myvolume myvolumegroup

congratulations! You now have a block device that you can use with any file system you want, e.g. btrfs or XFS:

mkfs.xfs /dev/myvolumegroup/myvolume
mount /dev/myvolumegroup/myvolume /mnt

should just work.

---

Remarks: are you sure you want this? 320 GB and 250 GB are probably not very new drives, and you get only (160+125)GB = 285 GB of extra storage by including these potentially very aged drives in here.

Using old drives is kind of contrary to the reason you'd want RAID1 to begin with: Data availability. Also, consider using RAID5 instead (man lvmraid might help you with that).