29

Can someone please explain to me, what the difference is between creating mdadm array using partitions or the whole disks directly? Supposing I intend to use the whole drives. Note, that they are with GPT.

Imagine RAID6 created in two ways, either:

mdadm --create /dev/md0 --level=6 --raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1

or:

mdadm --create /dev/md0 --level=6 --raid-devices=4 /dev/sda /dev/sdb /dev/sdc /dev/sdd

Is there a difference?

EDIT1: I mean in for example reliability or manageability or recovery operations on such arrays, etc.

EDIT2: I do use different manufacturer's drives. But that is not the question really. They all have the exact same size, I do get the point, that there may be manufacturers making drives slightly different sizes. But for the sake of this question, let's assume they are the very same size.

EDIT3: Sort of disclaimer: So far I am always partitioning the drives.

  • 2
    I've always been curious about this but never asked anyone. One difference I see is using the partitioned disks (sda1, etc) you get a little bit of less disk space (some KB) as GPT reserves some space for partition information. – GMaster Oct 31 '16 at 16:11
27
+50

The most important difference is that it allows you to increase the flexibility for disk replacement. It is better detailed below along with a number of other recommendations.

One should consider to use a partition instead of the entire disk. This should be under the general recommendations for setting up an array and may certainly spare you some headaches in the future when further disk replacements get necessary.

The most important arguments is:

Disks from different manufacturers (or even different models of the "same" capacity from the same manufacturer) don't necessarily have the exact same disk size and, even the smallest size difference, will prevent you from replacing a failed disk with a newer one if the second is smaller than the first. Partitioning allows you to workaround this;

Side note on why to use different manufacturers disks: Disks will fail, this is not a matter of a "if" but a "when". Disks of the same manufacturer and the same model have similar properties, and so, higher chances of failing together under the same conditions and time of use. The suggestion so is to use disks from different manufacturers, different models and, in special, that do not belong to the same batch (consider buying from different stores if you are buying disks of the same manufacturer and model). This is not uncommon that a second disk fail happen during a resotre after a disk replacement when disks of the same batch are used. You certainly don't want this to happen to you.

So the recommendations:

1) Partition the disks that will be used with a slightly smaller capacity than the overall disk space (e.g, I have a RAID5 array of 2TB disks and I intentionally partitioned them wasting about 100MB in each). Then, use /dev/sd?1 of each one for composing the array - This will add a safety margin in case a new replacing disk has less space than the original ones used to assemble the array when it was created;

2) Use disks from different manufacturers;

3) Use disks of different models if different manufacturers are not an option for you;

4) Use disks from different batches;

5) Proactively replace disks before they fail and not all at the same time. This may be a little paranoid and really depends on the criticity of the data you have. I use to have disks that have 6 months differences in age from each other;

6) Make regular backups (always, regardless if you use an array or not). Raid doesn't serve the same purpose of backups. Arrays assure you high availability, Backups allow you to restore lost files (including the ones that get accidentally deleted or are damaged by viruses, some examples of something that using arrays will not protect you from).

OBS: Except for all the non-neglectable rational above, there aren't much further technical differences between using /dev/sd? vs /dev/sd?#.

Good luck

| improve this answer | |
  • That's mostly not related... – user1133275 Nov 15 '16 at 23:48
  • Yet, I guess @Kalavan warning refers to use multiple partitions in the same disk as components of an array. That certainly wouldn't make much sense as it removes the redundancy. We know, though, that this was not the intention of your question, His warning makes sense anyway. – Marcelo Nov 16 '16 at 23:16
  • If managing the partitions manually seems to be a hassle, one should use sfdisk -d src | sfdisk tgtor sgdisk --backup && sgdisk --load-backup – andyn Jan 3 '19 at 10:15
  • Don't forget about Advanced Format 4K drives that benefit from Partition Alignment (which most modern large multi-TB drives have nowadays). For most drives, the first available sector is not zero, meaning using the whole drive would cause partition misalignment. – ryancdotnet Jan 3 at 4:48
8

Another important argument is that some mainboards may delete your RAID superblocks if you use whole disk devices and are not super careful with wiping them when adding disks to a RAID array that once were GPT devices.

I learned this the hard way, when my ASRock mainboard's UEFI rendered my RAID unusable:

To protect yourself against this, always run sgdisk --zap on a formerly GPT-formatted disk that you want to use for a whole-device RAID.

wipefs -a may also work to wipe all kinds of leftover partitioning info (not only GPT) before adding the device to the disk.


That said, just using partitions seems like a best practice to make this less likely to happen.

I never knew what the real arguments were for this best practice (beyond @Marcelo's answer explaining that it can deal with disks of slightly different sizes). Now I know.

By the way, if this happens to you, your data is not lost. You can most likely just sgdisk --zap the device, and then recreate the RAID with e.g. mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdc /dev/sdd (mdadm will tell you that it already detects past data, and asks you if you want to continue re-using that data). I tried this mutliple times and it worked, but I still recommend taking a backup before you do it.

| improve this answer | |
2

There are couple of posts citing problem with using the whole device as a RAID member instead of partition, one here. I can confirm that to be the case.

So, if you create RAID using the whole device (say, /dev/sda, /dev/sdb, ... instead of on /dev/sda1, /dev/sdb1, ...) the RAID device doesn't reassemble after a reboot even if you have configuration saved under mdadm.conf. There is also a possibility of mainboard firmware overwriting or removing the RAID superblock. In my setup, running mdadm --assemble --scan --verbose doesn't get me back my RAID setup and it seems to be the case of RAID superblock not being available.

| improve this answer | |
  • I tested RAID1 setup using 2x10GB drives inside VirtualBox. RAID was created on the whole device and it is reassembling fine after reboot. The problem could be applicable only to physical devices. – Lungten Jan 1 '19 at 5:36
1

For saving yourself from future headaches, use partitions. All drives fail, it's not a question of "if" but "when". If you are using whole drives, then after a drive failure, you must replace it with exactly the same size model. Even a few MB larger, due to different manufacturer, or process changes, whatever, will cause the drive replacement to fail.

Using partitions, when one of your current 2TB drives fails, you get a replacement 4TB (or whatever, just needs to be larger), then create the same 2TB partition size on it. Yes, you may be wasting space on that new large drive. Using the rest of that drive as a non-raid partition may be possible, but could affect performance in weird ways. After several drive failures and replacements, you will eventually replace all of your original 2TB drives, and now have the ability to extend your RAID array to use larger partition sizes.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.