Most ironclad way to make root installation redundant and maximize uptime? RAID, ZFS or something else?

Question

I would like to set up my desktop computer (which is actually a server for the KVM guests I do my actual work in) to have a redundant root installation. If one drive dies I want to quickly get back to work without doing a full restore from backup, nor a system reinstall and reset all my settings and preferences.

I thought that the way to do this would be RAID1, but the deeper I dig into it, the more I realize that RAID1 is not a 'set-it-and-forget-it' solution. Oh, and I want it to be UEFI boot.

Last time I tried a software RAID1 install (which I set up using the Ubuntu Server installer), something got corrupted and I ended up with a GRUB rescue screen and could not for the life of me figure out how to get it to boot from the mirror drive. For all I know, the boot sector on both was corrupted due to the corruption replicating between drives. Obviously this defeats the purpose of having a RAID1 boot for the purpose of decreased downtime. I was thinking that maybe I should put the EFI partition on a USB drive and keep it backed up for quick and easy replacement (while having the root partition in RAID1), but I am worried that I might now always know then the EFI partition has changed and therefore will not know when to back it up.

I was also thinking to do ZFS-on-root, in the thought that the bitrot protection and snapshotting might be more useful in preventing situations like the one above. But it seems that ZFS on root is not recommended for Ubuntu, and the status of ZFS on Linux in general seems to be in question now due to a certain Linux Kernel programmer's stated lack of tolerance for ZFS. I wonder if this might be a good approach but I know nothing about this whole MAAS thing and have no idea whether it is relevant to my use case.

The last thing I was thinking was to just do a regular one-drive install and then every week or so dd it to a spare drive, so that if disaster strikes I can at least recover my settings and installation from a week ago or less. But wouldn't dding an SSD every week be really hard on it?

I have found countless tutorials about RAID and ZFS, but so far have not found anything that clearly explains to pros and cons of my options with respect to the goal stated above. Advice or links to explanations would be greatly appreciated!

Answer 1

Depends on needs, but you have a few options. My personal choice is to use lvm mirroring on a root volume, and any other that is critical to my sanity (/home on my laptops and workstations).

As for backups, you could tarball up or rsync your stuff to a remote host, or if it's a bit simpler even use git (works wonders on /etc).

I used to just use mdadm to do mirror and stripes, and gave up and just use lvm since it's far easier to migrate things (swap drives, add drives, move to new host) than mdadm is.

Answer 2

Most Ironclad way & maximize uptime...

I think the question overall is a good one but will be problematic to answer given the current state of everything that makes up linux, and hardware. There can be more than one correct answer and then that could still be rationally argued. A better way to answer this might be a what not to do...

• understand your hardware, and use good equipment
• hardware RAID is better than software RAID (mdadm)
• I have experienced data corruption with software raid (mdadm) on JBOD storage units because of power dropouts, so I always opt for proven hardware RAID equipment when possible otherwise you take your chances. Don't use software raid if you are serious
• have some kind of uninterruptible power supply (UPS) to handle power outage and brown out scenarios- only need to keep running long enough to do graceful shutdown to preserve data
• understand RAID: redundant array of independent disks = it can be a set it & forget solution if done at the hardware level and done properly. Unless you need more than 2 disks to get storage capacity, stick with a simple RAID-1. one drive fails you're still running like nothing happened, otherwise use 3 disks as a RAID-6 with 2 disk fault tolerance if you are paranoid.
• understand RAID is not backup. So have 2 or 3 disks as RAID-1 or RAID-6 as your root partition, bios or efi boot loader boots that one disk (a raid volume). Have a separate disk... somewhere else, that is a mirror copy of what is on the two or three disk RAID volume that is active. When that gets corrupted (regardless of raid) you resort to restoring from whatever the backup was.
• have some understanding of disaster recovery and what context that means for your scenario; be able to fix and restore from any problem happening and know how long that will take you.
• Maybe NOT use revolutionary new file system that fundamentally changes the way file systems are administered which is the description of ZFS. My recommendation would be to stick with something proven such as EXT3 or EXT4 or XFS, and maybe BTRFS but only if the linux distribution supports it well. Problems often involve errors with the file system that can be fixed... if you understand the tools that are available for that file system.
• Pretty sure every linux distro supports EXT3 and XFS, EXT4 as well as BTRFS may not be on all distros unless using their latest version, and use a good linux distribution. For most ironclad my recommendation would be to lean towards one of the enterprise linux's... RHEL, SLES, CentOS where there is good support.