1

My current idea is to create 1x software RAID-6 with 4 member drives using mdadm.

Specifically, they would be 1TB HDDs on SATA in a small server.

System: Linux Debian Jessie 8.6.

That makes 2TB of space with 2TB for parity.

But I would like to have separated 1TB of space and another 1TB on some level.

I don't think it would be wiser to create 2 separate 2-drive RAID-1 arrays, or would it?

I'd rather create two separate ext4 filesystems on this one RAID-6 array.

Q #1) Does this look good or bad? Any problems I should take into account?

I would also like to have it with GPT partition table, for that to work, I am unsure how to proceed specifically supposing I would prefer to do this over terminal.

Q #2) Could you guide me how I should proceed in a few steps.

Thank you for your time.

EDIT1: This array will serve for the sole data only. No boot or OS on it.

EDIT2: RAID5 is totaly out of the question; RAID1 would be wasting space; RAID10 is great, but I need to opt for RAID6 due to the purpose of this array, two disks fail the array must be able to survive. Since I am limited by HW to 4 drives, there is no alternative to RAID6 that I know of. However ugly RAID6 slowdown may seem, it does not matter in this array.

4

In this answer, let it be clear that all your data will be destroyed on all of the array members (drives), so back it up first!


Open terminal and become root (su); if you have sudo enabled, you may also do for example sudo -i; see man sudo for all options):

sudo -i

First, we should erase the drives, if there was any data and filesystems before, that is. Suppose we have 4 members: sda, sdb, sdc, sdd:

pv < /dev/zero > /dev/sda
pv < /dev/zero > /dev/sdb
pv < /dev/zero > /dev/sdc
pv < /dev/zero > /dev/sdd

To double-check if there is nothing left behind, you may peek with GParted on all of the drives, and if there is any filesystem other than unknown, wipe it:

wipefs --all /dev/sda
wipefs --all /dev/sdb
wipefs --all /dev/sdc
wipefs --all /dev/sdd

Then, we initialize all drives with GUID partition table (GPT):

gdisk /dev/sda
gdisk /dev/sdb
gdisk /dev/sdc
gdisk /dev/sdd

In all cases use the following:

o Enter for new empty GUID partition table (GPT)
y Enter to confirm your decision
w Enter to write changes
y Enter to confirm your decision

Now, we need to partition all of the drives, but don't do this with GParted, because it would create a filesystem in the process, which we don't want, use gdisk again:

gdisk /dev/sda
gdisk /dev/sdb
gdisk /dev/sdc
gdisk /dev/sdd

In all cases use the following:
n Enter for new partition
Enter for first partition
Enter for default of the first sector
Enter for default of the last sector
fd00 Enter for Linux RAID type
w Enter to write changes
y Enter to confirm your decision

To triple-check if there is nothing left behind, you may peek with GParted on all of the newly created partitions, and if they contain any filesystem other than unknown, wipe it:

wipefs --all /dev/sda1
wipefs --all /dev/sdb1
wipefs --all /dev/sdc1
wipefs --all /dev/sdd1

You can examine the drives now:

mdadm --examine /dev/sda /dev/sdb /dev/sdc /dev/sdd

It should say:

(type ee)

If it does, we now examine the partitions:

mdadm --examine /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1

It should say:

No md superblock detected

If it does, we can create the RAID6 array:

mdadm --create /dev/md0 --level=6 --raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1

We shall wait until the array is fully created, this process we may watch with:

watch -n 1 cat /proc/mdstat

After creation of the array, we should look at its detail:

mdadm --detail /dev/md0

It should say:

          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

Now we create filesystem on the array, if you use ext4, this is better to be avoided, because of ext4lazyinit would take noticeable amount of time, hence the name, "lazyinit", therefore I recommend you to avoid this one:

mkfs.ext4 /dev/md0

Instead, you should force a full instant initialization with:

mkfs.ext4 -E lazy_itable_init=0,lazy_journal_init=0 /dev/md0

By specifying these options, the inodes and the journal will be initialized immediately during creation, useful for larger arrays.

If you chose to take a shortcut and created the ext4 filesystem with the "better avoided command", note that ext4lazyinit will take noticeable amount of time to initialize all of the inodes, you may watch it until it is done, e.g. with:

iotop

Either way you choose to make the file system initialization, you should mount it after it has finished its initialization:


We now create some directory for this RAID6 array:

mkdir -p /mnt/raid6

And simply mount it:

mount /dev/md0 /mnt/raid6

Since we are essentially done, we may use GParted again to quickly check if it shows linux-raid filesystem, together with the raid flag on all of the drives.

If it does, we properly created the RAID6 array with GPT partitions and can now copy files on it.

See what UUID the md filesystem has:

blkid /dev/md0

Copy the UUID to clipboard.

Now we need to edit fstab, with your favorite text editor:

nano /etc/fstab

And add add an entry to it:

UUID=<the UUID you have in the clipboard>                /mnt/raid6              ext4    defaults        0 0

You may check if it is correct, after you save the changes:

mount -av | grep raid6

It should say:

already mounted

If it does, we save the array configuration; in case you don't have any md device yet created, you can do simply:

mdadm --detail --scan >> /etc/mdadm/mdadm.conf

In case there are arrays already existent, just run the previous command without redirection to the conf file:

mdadm --detail --scan

and add the new array to the conf file manually.

In the end, don't forget to update your initramfs:

update-initramfs -u

Check if you did everything according to plan, and if so, you may restart:

reboot --reboot
0

If you create the RAID arrays over 4 raw block devices instead of 2 x 4 partitions, that means all RAID recovery operations may necessarily operate on the whole devices, and vice versa.

So for example if you expect the disks to eventually start developing I/O errors on the latter half of them, with the arrays over partitions that would mean that only one of the arrays would notice, while the other would continue untouched, at least until the damage spreads into its half. This might provide you with some temporary flexibility, or an absence of I/O slowdown.

On the other hand, as soon as you start intervening, you have to take out the whole physical disk for replacement, so you necessarily have to degrade both arrays at some point. Also, with SSDs, it's apparently become more common that the whole disk fails, so both arrays might be affected by such events anyway.

There is nothing to be said about the details of the process that can't be found in typical partitioning and mdadm documentation.

  • @VlastimilBurian please ask specific questions, rather than generic "show me how to do everything" questions. Cf. meta.stackexchange.com/questions/18242/… – Josip Rodin Oct 23 '16 at 9:03
  • You can replace disks without degrade using mdadm --replace. You just need a free sata port for 5 disks in the system for the duration of the resync. – frostschutz Nov 1 '16 at 16:49
  • @frostschutz surely that assumes the array didn't get into a degraded state automatically by way of unrecoverable errors? – Josip Rodin Nov 2 '16 at 15:58
  • That's what you described... multiple partitions per disk, only one degraded. Or did I misunderstand you? In that situation you don't "necessarily have to degrade both". You can --replace the not-degraded one. Same if the errors turn up in SMART selftest but RAID doesn't know about it yet. – frostschutz Nov 2 '16 at 16:11
  • @frostschutz that's true, but the original question talked of a small server, and specifically about four disks, so I had assumed that those are constraints, not just arbitrary variables. – Josip Rodin Nov 2 '16 at 20:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.