Setting up NixOS with lvmcache and LUKS-on-LVM

Question

I'm trying to install NixOS 19.09 on a notebook. My disk setup is a 30GB SSD and a 700GB HDD. I decided that I want to use the SSD as cache for the HDD, and also that I want the entire root partition to be encrypted.

My partition setup therefore looks like this:

• /dev/sda - SSD
  • /dev/sda1 - boot partition, UEFI bootable flag, formatted with FAT, unencrypted
  • /dev/sda2 - to be used for the cache
• /dev/sdb - HDD, unpartitioned, to be used as the main disk with LVM

I formatted /dev/sda1 with mkfs.fat, I set up a volume group vg that included /dev/sda2 and /dev/sdb, and I then followed the guide that you can find in the lvmcache manual page. After creating a single cached volume called hdd0, I ran cryptsetup on it, so that I could achieve a LUKS-on-LVM scheme. The entire cached hdd0 is encrypted with LUKS2.
Then I unlocked hdd0 again using cryptsetup, mapping it to /dev/mapper/root. I mounted /mnt on it, created /mnt/boot and mounted /dev/sda1 there. After that, I generated a NixOS config and basically left it at default, with the notable changes of adding dm-cache to boot.initrd.kernelModules, and setting boot.initrd.luks.devices."root".preLVM to false. Then, I ran nixos-install successfully and rebooted.

The system managed to successfully run GRUB, but after that failed to find the root volume. It provided an error message of:

device-mapper: table: 254:3: cache: Error creating cache's policy  
device-mapper: reload ioctl on (254:3) failed: Invalid argument

I'm assuming it has a problem with detecting the entire hdd0. I tried booting it from the live installation USB stick, and there you have to enable dm-cache using modprobe and then run

# lvchange -ay vg

to get hdd0 to be active and visible.

Did anyone try a similar setup? How can I get it to boot automatically? Or is the setup flawed conceptually and can never work?

PS: I hope the question is detailed enough, it's my first time asking here.

Answer 1

Here's my conf that I got to work, today, after some headbangery:

boot.initrd.availableKernelModules = [ "nvme" "xhci_pci" "ahci" "ehci_pci" "usb_storage" "usbhid" "sd_mod" "sr_mod" ];
  boot.initrd.kernelModules = [ "dm-cache" "dm-cache-smq" "dm-cache-mq" "dm-cache-cleaner" ];
  boot.kernelModules = [ "kvm-amd" "dm-cache" "dm-cache-smq" "dm-persistent-data" "dm-bio-prison" "dm-clone" "dm-crypt" "dm-writecache" "dm-mirror" "dm-snapshot"];

  boot.initrd.luks.devices = {
    "decrypted" = {
      device = "/dev/mapper/vg-crypt";
      allowDiscards = true;
      preLVM = false;
    };
  };

The main trick was to have "dm-cache-smq" in modules - without it, I was getting a similar error message like what you're getting.

You'll have an ugly warning message about cache_check missing - to get rid of it, add this too:

services.lvm.boot.thin.enable = true;