Here's what I'm trying to accomplish, per-VM virtual disk image. (Win10 and Ubuntu VMs as "desktop" OSes, among other minor headless VMs):

  1. At-rest encryption (w/ thumbdrive key)
  2. Transparent compression
  3. Automatic snapshots
  4. Minimal write amplification to underlying devices
  5. (Optional) redundancy aka RAID

I know exactly how to accomplish 1, 2, 3, and 5 - with ZFS. But crucially, not #4, if it's even possible. I understand that the write-amplification can be downright ghastly with ZFS backing storage for VM images, even if you get the stacked filesystems, including underlying ZFS itself, configured just right and everything optimally aligned. (And the penalty for getting it wrong, disastrous. Or so I've read.)

I also know first-hand how raw-mapped devices, or IOMMU passthrough, can perform near-natively, with no write-amplification. (And I realize that with virtual disk images, some amplification is unavoidable.) But the whole point is to transition completely to portable, snapshottable, easily backed-up, high-performance VMs with hardware-passthrough GPU. I've been doing almost all of that for over ten years, with regular 'ol virtualbox, .VDI disk images on ext4, and regular virtual display "acceleration". But now the tech is easily available to take it to the next level. (Except obvious ways to approach the virtual storage goals.)

Any direct experience and/or ideas? I'm open to any tech, FLOSS host OSes (I'm not married to Ubuntu), etc. My bucket of virtual disk ideas in terms of backing storage per VM, currently looks like:

  • Btrfs RAID-1 at lowest common level, for compression and per-VM snapshots (but no stable encryption)
  • ZFS RAID-Z1 with LZ4 and per-VM snapshots and at-rest encryption

Then of course on top of that, some QEMU container, + guest filesystem (e.g. NTFS, ext4, etc.).

It's on new hardware, fresh Ubuntu 20.04 host w/ QEMU, KVM, and libvirt. 32 threads, 128GB RAM. Available unformatted drives:

  • 3 x 1TB NVMe (different brands)
  • 1 x 1TB SATA SSD
  • 1 x 2TB SATA SSD

Any ideas appreciated!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.